National deputy leader Paula Bennett is accusing Speaker Trevor Mallard of a "serious lapse in judgment" for calling a parliamentary worker the equivalent of a rapist.
But Prime Minister Jacinda Ardern is standing by Mallard, saying she continues to have confidence in him.
The parliamentary worker, who was stood down last week as a historical allegation of assault was reopened, has hit back, saying he is the victim of Mallard's "slanderous" comments.
"I never thought I would ever find myself in this situation, it's not who I am, I'm thoroughly devastated," the employee told Newstalk ZB political editor Barry Soper.
"I would like to be able to return to work to clear my name and I expect, at the very least, an apology from the Speaker for labelling me as a rapist which I most certainly am not."
It followed last week's release of the Francis review that showed a toxic culture of bullying and harassment at Parliament.
The following day, Mallard said in a Radio NZ interview that his reading of the report was that a man responsible for three serious sexual assaults mentioned in the review, and that he believed the man was still working at Parliament.
He added that the incidents were tantamount to rape and, when the man was stood down, that a threat to safety had been removed from the premises.
Mallard has refused interviews yesterday and today.
Bennett said she did not know any details of the case and could not say whether the man should have been stood down.
But she slammed Mallard for his "reckless" comments.
"There has been poor judgment and I think it's at the serious end ... It was extraordinary.
"He has caused a whole lot of problems, and quite frankly, I question his judgment on this."
Bennett took part in meetings with Mallard following widespread concern about Mallard's rape comments, and she said Mallard appeared to know nothing more than what was in the Francis report.
"What concerned us a lot was that he was then, in our mind, adding two and two together and coming up with 10.
"I certainly challenged him on the use of the language of rape. He certainly stood by that. I just didn't know anything further and he couldn't verify whether it had been or not.
"To then recognise that that might not actually have been what had happened, for me, was really alarming."
She said the man's claims that he was investigated, and exonerated, raised further questions that Mallard had the responsibility to answer.
"I don't think you get to hide behind an employment matter at this point. He has made comments in the past, very recently, and the public and certainly this Parliament has a right to have answers."
The man said that his family was dumbfounded and the harassment claims unsubstantiated.
He claimed the three allegations were related to hugging a colleague, complimenting another colleague on her hair, and kissing another on her cheek as he said goodbye to her after she visited him and his wife for tea.
However, the first complainant alleged he hugged her from behind, pushing his groin up against her, and that he was staring at the breasts of the woman whose hair he complimented. He believes the third complainant was put up to lodging the complaint by someone else.
Ardern stood by Mallard this morning, but would not comment on an individual's employment matter.
"Ultimately some serious allegations were made in that report that point to the fact that we need to make sure that Parliament, generally, is a safe place to work."