He said Mallard had removed Jesus' name to be more inclusive – "but the effect has been the polar opposite".
In fact, he said the move was a "blatant and unbridled misuse of power by one man".
By removing Jesus' name from the prayer, what is left is a "politically correct, watered-down, inoffensive, one-size-fits-all God – that's the Speaker's idea of God," Smith said.
Clearly many in the crowd agreed – one banner read: "Dishonourable Judas Mallard".
Many other signs implored Mallard to add Jesus back into the prayer.
As it stands, the parliamentary prayer contains a reference to God – a spokesman for Mallard said this was more inclusive to all religions.
Smith also took aim at Prime Minister Jacinda Ardern during his address to the rally.
He said that earlier this year, Ardern declared New Zealand had no official or established religion.
This drew boos and jeers from the crowd.
"Seriously? Are you kidding me? I think someone lives in a vacuum," Smith said.
Also among the speakers was National MP Alfred Ngaro, who told The Herald he had been supportive of the cause ever since Jesus' name was removed from the prayer.
"Historically, the footprint of faith in our country goes all the way back 160 years."
He said the very first speech and debate in Parliament was about prayer.
A spokeswoman for the Speaker said he would not be commenting further.
Speaking to media this morning, National Leader Simon Bridges said Mallard should have consulted MPs before removing the reference to Jesus.
"I think if he did, he might have come to a different position – ultimately we don't know but I don't think it was his decision to make alone."
Deputy Prime Minister Winston Peters agreed.
He said the decision to remove Jesus' should have been decided by parliamentarians and not the Speaker.
"I think he should have first consulted all of Parliament."
