The tight vote means 40 per cent of the Liberal Party do not want Turnbull to continue as leader and has increased the chance of a rushed election before Christmas.
Mr Dutton put his hand up to challenge Mr Turnbull for the top job after the PM's risky move to take control of the party.
The deputy leadership was also declared vacant, but Julie Bishop was the only person nominated so hung on to her role.
Mr Turnbull offered for Mr Dutton to remain in cabinet and keep his job of Home Affairs, but he declined the offer, The Daily Telegraph reported.
Mr Dutton visited Mr Turnbull in the ministerial wing of Parliament House shortly before the formal meeting.
Sky News political editor David Speers said the fact that 35 of his party members voted against Mr Turnbull was a "humiliation" for him.
"In Parliament, he's going to be standing there in front of a backbench, a third, nearly half of whom don't support him," he said.
"Incredibly difficult to see how Malcolm Turnbull can recover from this."
While the challenge over the prime ministership was ostensibly about energy policy, the true source of the leadership tensions goes much deeper.
Mr Dutton was seen as an alternative leader who could better represent the conservative wing of the party, which is not interested in locking Australia into ambitious carbon emissions cuts.
Mr Turnbull has reportedly said there would be no early election and he had not considered it.
Former deputy PM Barnaby Joyce has reportedly reached out to Mr Turnbull offering his "help", according to Sky News.
Chief government whip Nola Marino announced the close margin in the votes after the tense party room meeting.
"It is always orderly in our party room," Ms Marino said.
"He (the PM) thanked the colleagues for their support."
Commentators have declared the troubled result for the PM as a "mortal wounding", securing support of more than 40 per cent of the party room.
Mr Dutton's resignation means Mr Turnbull will have to reshuffle his cabinet, with commentators predicting a second leadership ballot in the weeks or months ahead.
The move also raises questions over whether any of Mr Dutton's allies could stand down.
If he had won today's spill he would have become Australia's sixth prime minister in just eight years.
Mr Turnbull won his spill against Tony Abbott in September 2015 when he won by a margin of 10 votes, 54 to 44.
Julia Gillard won her first challenge from Kevin Rudd who was only about to collect support from about 30 per cent of his colleagues. Ms Gillard was defeated in the second spill.
Bob Hawke won his first ballot with 60 per cent of the vote to lose the second challenge.
ABC's Patricia Karvelas said the result was devastating for Mr Turnbull and could spark another leadership challenge in days.
"I've had contact with some of the people doing numbers and building support for Peter Dutton who are saying that as close as Thursday, yes — it's Tuesday — that's in two days, they may have another shot," she said.
"Because they feel like the Prime Minister didn't give them time, that it wasn't necessarily a very fair process for them to build their numbers," she said.
"It is devastating for the Prime Minister. There's no other way of viewing it."
ABC political journalist Barrie Cassidy said Mr Turnbull adopted his "tactic" for everyone to see.
"Why did he declare the position vacant today? And surely, that is because he knew that three weeks from now, there would be two more terrible Newspolls, it would give Peter Dutton lots of time to make all of those calls and to round up the numbers, so he decided to strike today," Cassidy said.
"But then by doing that, he's just exposed the weaknesses of his position. You can't … even putting aside who wins this particular ballot, you can't claim to have the 35 people voting against you."
Mr Abbott tweeted a clarification of reporters earlier being briefed that MP Warren Entsch was getting stuck into him in the party room, as it happened. There were reports Mr Entsch's speech was met with some claps.
"Unlike too many of my colleages, my practice is to take responsibility for what I think and say," he said.
"The put the Entsch intervention into context, I had just said to the party room that exhorations from the leadership group about loyality and united were all very well but united has to be created and loyalty has to be earned. They can't just be demanded."
Nationals frontbencher Darren Chester said he encouraged his colleagues to back Mr Turnbull to finish the job he started.
"Now it's time to back the Prime Minister, let's keep getting the job done," he said.
"Malcolm Turnbull continues to lead the Coaltion to the next election....and will win.
Asked whether frontbenchers who did not support Mr Turnbull should resign, Mr Chester said they should consider their future.
"Individual front and back benchers make deicions about their career every day of the week," he said.
"I would simply say to my colleagues the circus has to stop.
"He's won the right to contunue as PM and that should be the end of the matter."
Mr Chester ruled out the second strike theory being floated.
WHAT LED TO THE VOTE?
Malcolm Turnbull's support in the Liberal party room has collapsed in recent weeks, with poor polling and his ambitious energy policy causing leadership tension.
Mr Turnbull gutted two of his own signature policies in the past 24 hours in an effort to save himself, removing the carbon emissions target from the National Energy Guarantee (NEG) and depriving the Big Four banks of his company tax cuts.
A senior source told The Daily Telegraph Mr Turnbull was "in panic mode" and "clearly rattled" amid reports Mr Dutton was considering launching a challenge.
Mr Turnbull has previously cited Tony Abbott's 30 straight defeats in the poll as one of the main reasons for toppling the former prime minister.
One of Mr Turnbull's loyal supporters, Defence Industry Minister Chris Pyne, steadfastly denied a challenge was imminent during an interview on Sky News this morning.
Mr Pyne said he had spoken to Mr Dutton daily and he was "loyal and supportive" of Mr Turnbull.
"This instability, this constant revolving door of the leadership, is not good for the country," he said.
"I have spoken to him directly and he has told me he supports Malcolm Turnbull.
"If you have different information directly from the horse's mouth, say so. If it's just other people talking on Peter Dutton's behalf then they are cowards."
Mr Pyne said people would be "horrified" if the government changed leaders again.
"I think the Australian public would throw up their hands and say you guys in Canberra are playing games," he said.
WHAT'S NEXT?
* Malcolm Turnbull must reshuffle his ministry, with a number of frontbenchers expected to go to the backbench.
* The prime minister must rebuild bridges with the 35 Liberal MPs who voted against him.
* With the corporate tax cuts laws likely to fail in the Senate and the National Energy Guarantee shelved, Turnbull needs a fresh economic agenda. * Peter Dutton's forces could consolidate and have another crack, as has occurred in past leadership spills.
* Parliament wraps up for the week on Thursday and does not return until September 10.
* The federal election is due by May 2019, but Turnbull may be tempted to call it within weeks in a bid to sharply focus the minds of Liberal MPs, unify the party and ensure no further bloodletting.
* The PM heads to South-East Asia next week for trade and security talks.
* Labor has led 38 Newspolls in a row, but Turnbull has maintained a strong lead as preferred prime minister.
WAS DUTTON ELIGIBLE?
Meanwhile, a report late yesterday afternoon cast some doubt over Mr Dutton's right to sit in parliament.
Constitutional law experts told Ten the Home Affairs Minister could be disqualified from parliament over his business interests, Ten Eyewitness News reported.
In his parliamentary register of interest, Mr Dutton lists himself as a beneficiary of RHT Family Trust — along with his wife and children.
According to ASIC documents, the business owns two child care centres in northern Brisbane.
Under a law change Mr Dutton voted for, those centres have been receiving direct subsidies from the government since July 2 this year.
But Ten notes that, under Section 44(v) of the Constitution, any person with "any direct or indirect pecuniary interest with the Public Service of the Commonwealth" is disqualified from Parliament.
In April last year, the High Court ruled that then Family First Senator Bob Day was not validly elected to parliament, due to his breaching Section 44.
The Commonwealth had leased the office space at a building he owned in order to house his electorate office, the ABC reported.
Mr Day transferred the property to a family friend, but documents revealed there was an agreement that proceeds from the rent would go to a company he owned.
A spokesman for Mr Dutton told Ten: "Mr Dutton's legal advice clearly states there is no breach of Section 44."
But, if the case is referred by parliament, it could become a matter for the High Court to decide.
Meanwhile Hugh Riminton, the journalist who broke the story, has dismissed suggestions it was leaked to him by a Liberal Party member.
"We've been watching pecuniary interest matters with federal parliamentarians for a long time," he told The Project, noting his coverage of a similar story about Turnbull backbencher Barry O'Sullivan last year.
"The timing of all this talk of leadership issues is coincidental," said Hugh. "Many people won't believe that but it's true."
