Drawing a parallel with the last days of the apartheid regime in South Africa he warned: "If the day comes when the two-state solution collapses, and we face a South African-style struggle for equal voting rights (with Palestinians) ... then, as soon as that happens, the state of Israel is finished."
Israel is sensitive to any comparison to formerly apartheid South Africa, but Olmert has aired such views before. When he was deputy prime minister under Ariel Sharon four years ago, he favoured a withdrawal from most of the territories taken in the 1967 war that would leave Israel with a "maximum" number of Israelis and a "minimum" of Palestinians.
Olmert also warned about the loss of support of the Jewish diaspora once the question became framed in terms of one man, one vote.
"The Jewish organisations, which were our power base in America, will be the first to come out against us," Olmert told Haaretz, "because they will say they cannot support a state that does not support democracy and equal voting rights for all its residents."
Olmert pointed out that he had said similar things in an interview he gave four years ago.
"Since then, I have systematically repeated those positions," he said, adding that people "will say I'm having problems and that's why I'm trying to do [a peace process], but the facts must be dealt with justly."
Jews form a solid majority inside Israel proper, making up roughly 80% of the population of 7 million. However, if the West Bank and Gaza are included, Arabs already comprise nearly half the population of the region.
To ensure that Israel retains its solid Jewish majority, Olmert supports a large withdrawal from the West Bank and parts of east Jerusalem, following Israel's withdrawal in 2005 from Gaza.
Although Olmert and Abbas agreed to restart negotiations with the goal of finalising a deal by the end of 2008, there is much scepticism that these two politically weak leaders can push through an agreement by next year.
Two polls published in Israeli newspapers showed the Israeli public to be hopeful for a solution, but sceptical about their leaders.
The polls, conducted by the Dahaf Institute and Dialog agency, found that less than one in five Israelis believe the Annapolis conference, hosted by George Bush, was a success, and more than 80% of the public thinks the Israeli and Palestinian leaders will not meet their goal of reaching a deal in 2008.
Still, the Dialog poll said 53% of respondents support a final peace deal creating a Palestinian state alongside Israel.
In his interview with Haaretz, Olmert said the Annapolis conference "met more than we could have defined as the Israeli expectations, but that will not absolve us of the difficulties there will be in the negotiations, which will be difficult, complex, and will require a very great deal of patience and sophistication."
Olmert, said Israel now had a partner in Abbas, albeit a weak partner.
"He is a weak partner, who is not capable, and, as Tony Blair says, has yet to formulate the tools and may not manage to do so," Olmert said.
"But it is my job to do everything so that he receives the tools, and to reach an understanding on the guidelines for an agreement. Annapolis is not a historic turning point, but it is a point that can be of assistance."
A big problem for both Abbas and Olmert is Gaza, which has been under control of Hamas militants since they seized power from Abbas's Fatah faction in June. Hamas has denounced Abbas as a traitor and says he has no right to speak on behalf of the Palestinians.
In the latest violence in Gaza, Israeli planes killed four Hamas fighters in two strikes in the south of the Gaza Strip. An Israeli army spokeswoman said in one incident, planes attacked people wearing military style clothing in an area from which rockets had just been fired towards Israel.
In the second, planes killed two Hamas security men who planted explosives near the fence Israeli forces maintain around the coastal enclave.
Hamas and medical officials said the air strikes were carried out in the Khan Younis area.
There have been frequent clashes between Israel and Hamas militants since they seized power in June. Hamas has been boycotted by the west because it does not recognise Israel and Gaza, with its 1.5 million inhabitants have been isolated economically since the Hamas takeover.