But it is understood that his party is upset about Mallard's handling of Parliamentary Question Time and his treatment of MPs on the National side.
A source said Mallard, who must be a neutral referee, had been condescending and disparaging towards National's senior MPs.
In one case, he especially angered the party by deducting five questions at a time from Brownlee.
Brownlee said his party had previously expressed frustration at the way Mallard removed questions from the Opposition, but said it's non-attendance at the committee was "not necessarily" related to this.
"I think it is no secret we see that as an unfair way of dealing with matters that annoy him," he said.
Mallard said no one had directly raised any concerns with him. For that reason, he would not say whether he would change the way he ran Question Time.
"I won't comment on hypotheticals," he said.
The National MPs' absence means the committee cannot make some of its determinations at private meetings, and must instead make them publicly in the debating chamber.
"If there's not a Business Committee determination and the Government wants to do it, the Government passes the motion through the House," Mallard said.
"If the Government does not want to do it, then it just won't do it. It can take House time to do [these] things, it's always quicker not to."
One of the items on the agenda was a decision on who should replace former National MP Jonathan Coleman on the committee.
Politik reported that Labour planned to pass a motion to replace Coleman with a Labour MP, which would give the Government a majority on the committee.
Brownlee said that would be an "appalling" move.
