Ministers Julie Bishop and Malcolm Turnbull, seen as the most likely leadership contenders, have reportedly pledged not to challenge.
But backbencher Mal Brough, a Howard-era minister, has been urged to, Fairfax reported on Saturday.
Senator Brandis dismissed backbench rumblings as "a few anonymous voices and a couple of people who have been prepared to put their head up".
"(Mr Abbott) has the strong and unanimous support of his cabinet and the overwhelming support of his party room," he said.
Nevertheless, the Queensland senator conceded there had been "some blips" in the prime minister's judgment - notably last Monday's decision to give Prince Philip a knighthood.
That led to what Senator Brandis labelled the government's very worst week.
"But they were errors in judgment about fripperies," he said.
"If he makes the wrong call on something as ephemeral as whether or not Prince Philip should be a knight of the order of Australia, heavens above, that is so much a 10th order issue."
He was frustrated the knighthood had turned into a "dangerous distraction" which caused Campbell Newman's government to lose momentum in the final week of the Queensland election.
Mr Newman lost his seat on Saturday and Annastacia Palaszczuk's Labor team looks set to win government after one term in opposition.
While Mr Abbott didn't say what lessons he would take from the Queensland experience, Senator Brandis found two.
First was a new volatility of Australian voters and secondly that reform is hard to achieve without the public joining a government's journey.
The devastating Queensland result comes as a new poll shows the government trailing Labor 57 to 43 per cent on a two-party preferred basis, compared to 55 to 45 per cent in December.
The prime minister's personal approval rating is wallowing at 27 per cent, according to the Galaxy Poll published in NewsCorp's papers on Sunday.
Senator Brandis had anticipated the polls might be worse than this.
But opposition leader Bill Shorten said he would be "staggered" if the coalition dumped Mr Abbott, given their fuss about Labor's knifing of Kevin Rudd and Julia Gillard.
Policies such as the Medicare co-payment, university fee deregulation and pension cuts had caused the coalition's unpopularity, he said.
"If they think it is the salesman, not what they're selling, then they will have learned nothing from the Victorian, the Queensland elections and the South Australian by-election," he told ABC television.
Greens leader Christine Milne said Queenslanders wanted to kick out both Mr Newman and Mr Abbott.
"It's not just Tony Abbott, it's the Abbott government policies that are the problem," she told reporters in Sydney.
"They have to reject the cruel and arrogant way that they have been leaning on the poor."
ACTU President Ged Kearney said Queenslanders had sent Tony Abbott and the Queensland LNP a clear message that Medicare, wages and workers' rights are not up for grabs.
-AAP
