He called for the MP to stand, withdraw and apologise.
When no one came forward, Mallard said he would listen back to the tape. He would not repeat the remark but said it was "very sexist".
The culprit, who was sitting somewhere behind National leader Simon Bridges, has still not been identified.
Bridges today said he wasn't sure the comment was made in the first place.
Asked by Newstalk ZB's Heather Du Plessis-Allan who had made the remark, Bridges said: "I was sitting there, I didn't hear it. I'm closer to the supposed people who might have said it. I've informally gone around some of the people who supposedly, in the line-up of culprits or possible culprits, no one's told me they said it, so was it even said?"
Bridges said the Speaker had called Stuart Nash a "silly child". Winston Peters had called Nathan Guy a "fool".
"I'm not excusing this sort of stuff, I'm simply saying to you it is the cut and thrust of Parliament."
NZ First Ron Mark was censured today for telling National MP Amy Adams to "settle petal" in the House.
Amy Adams: She just said that — she just said that.
Ron Mark: Settle petal, settle.
Speaker: Order! I think the member's just sort of evened it up for Mr Brownlee, and it's made my task slightly easier. He will stand, withdraw, and apologise. I just want to make it clear that that sort of sexist remark is not going to be accepted in the House. Stand, withdraw, and apologise.
Ron Mark: I withdraw and apologise.
