But he said that in the past, "Mallard has gotten away with saying far worse".
After watching Mallard clash with fellow National MP Paul Goldsmith yesterday, Bridges stormed out of the House with Goldsmith.
A number of MPs and reporters heard Bridges call Mallard a "twat".
Mallard said he did not hear him say this and, as such, he considered the matter closed.
But Bridges said the Opposition is frustrated with Mallard, who he said pulls National up on many issues when they are just trying to do their jobs – "it's frustrating".
"I've made enough frank calls for this week so let's leave it on this one," Bridges said when asked to reflect on Mallard's refereeing of the House.
Speaking to reporters after the incident yesterday, Leader of the House Chris Hipkins said Mallard was not acting unfairly to National.
He said National should spend a "bit less time yelling at the referee and a bit more time refining their own game".
On Bridges' "twat" comment, he said: "Ultimately, I think that New Zealanders expect us to focus on bigger issues than that".
Meanwhile, Bridges has denied that he was reprimanded by his leader, Judith Collins, after he called Police Commissioner Andy Coster a "wokester" on Tuesday.
Collins had said that she had talked to Bridges about this.
But Bridges this morning said she did not tell him off.
"I've great, great respect for the leader of the National Party, but I only get tellings off from one person. And that's my wife."