"I will do everything I can as Prime Minister to steady the ship over the coming weeks and months but I do not think it would be right for me to try to be the captain that steers our country to its next destination," he said.
The betting agency Paddy Power installed former London mayor Boris Johnson as their 11/8 favourite to replace Mr Cameron. Mr Johnson was followed in the betting by Theresa May and Michael Gove.
The British exit - "Brexit" - campaign passed the winning post in the EU referendum around 6am in Britain (5pm NZT).
UK Independence Party leader and Brexit campaigner Nigel Farage declared it to be "independence day" as the Leave campaign racked up votes across England and Wales - defying the verdict of London, Scotland and most of the big cities.
While England voted overwhelmingly for Brexit, Scotland and Northern Ireland backed Remain.
London backed Remain but the turnout was lower than expected because of bad weather.
In a move to calm the markets, the UK Chancellor George Osborne was heading up talks between the Treasury, Bank of England and European Central Bank.
Mr Cameron sought to reassure EU migrants in the UK and expats on the continent that their circumstances would not change - at least in the short term. The EU is due to hold a summit in Brussels where Mr Cameron is expected to formally notify them of Britain's intention to leave, triggering Article 50 of the Lisbon Treaty - the legal process for quitting the 28-nation bloc.
That starts the clock on a two-year period during which new arrangements are made in trade, justice and reciprocal visas.
Prime Minister John Key said it had been "a privilege" to work alongside Mr Cameron, who he called "a friend." Mr Key said the impact of the vote on New Zealand's trade arrangements with Britain and the EU would be limited.
"We remain committed to the launch of formal negotiations on an EU FTA, and will be working with the UK as they go through the process of leaving the EU to put in place new arrangements," Mr Key said.
European Council leader Donald Tusk led the stunned continent's reaction to Brexit insisting: "What doesn't kill you makes you stronger".
Mr Tusk said Britain's "historic" vote should not provoke "hysterical reactions" and urged the rest of the bloc to "keep our unity as 27".
But Europe's far-right parties hailed the vote as a victory for their own anti-immigrant and anti-EU stance and vowed to push for similar referendums in their own countries. The Dutch anti-immigration leader Geert Wilders called for a referendum on the Netherlands' EU membership.
"I think it's historic," he told Dutch radio.
France's Front National leader Marine Le Pen said: "Like a lot of French people, I'm very happy that the British people held on and made the right choice.
Mr Farage said the day would go down in history as "our independence day".
In a remark that could prove controversial after Labour MP Jo Cox was shot dead last week, Mr Farage said the country was separating from the EU "without a single bullet being fired".
- Agencies
