Budapest's move was announced after the sports-mad Orban, a big champion of the bid, held talks with Budapest mayor Istvan Tarlos and the Hungarian Olympic Committee on Wednesday evening.
"Budapest will formally ask the city council to withdraw Hungary's candidacy in agreement with the government," Tarlos was quoted as saying by the MTI news agency after the meeting.
The mayor had previously said he could not "ignore the will of the people" in light of the referendum push.
Bid chairman Balazs Furjes said it made no sense to continue the effort, unless widespread political support was restored, describing Budapest's bid as a "lame duck" in the current political climate.
Budapest became the eighth city to withdraw from the bidding, prompting calls for a permanent Olympics location to avoid the rigmarole of selecting a fresh city every four years.
The bid was dealt a fatal blow after a group of young activists collected more than a quarter-of-a-million signatures of Budapesters within a month to hold a vote on the bid.
The Momentum Movement (MoMo) announced last week that it had garnered some 266,000 signatures, almost double the threshold required to trigger a ballot.
Within 24 hours, the Hungarian firm in charge of organising the event suspended its operations, while awaiting a decision.
The so-called "Nolimpia" drive had been aided by several opposition parties critical of Orban, prompting the government to accuse them of turning the matter "into a political party affair".
The strongman had backed the bid, launched in 2015, as a reward for his country's rich Olympic record - only nine countries have won more medals in the history of the Games.
Supporters said that Budapest, which, unlike Paris or Los Angeles, has never hosted the Games, was more suited than its rivals to the IOC's low-cost 'Agenda 2020' strategy.
But critics, fearing spiralling costs and corruption, said taxpayers should decide how their money is being spent.
Recent polls also showed clear majorities in favour of withdrawing the bid. Opponents had tried several times to force a referendum on the issue, but were blocked by either city hall or the supreme court.
"The people should have been asked two years ago exactly to avoid such a situation," said MoMo leader Andras Fekete-Gyor.
Budapest's city council, which holds a majority of Fidesz MPs, now needs to approve the bid's withdrawal.
"I am sorry to see a dream dissolve," the chief of the Hungarian Olympic Committee, Zsolt Borkai, said at a council meeting earlier Wednesday.
Paris-2024 choose not to comment on the development from the Hungarian capital.
