That prompted a revolt from MPs.
One Opposition MP after another stood to make a point of order, urging the Speaker to make Mr Key apologise.
Several of them, including Greens' Metiria Turei, Jan Logie, Catherine Delahunty, and Labour's Poto Williams revealed that they were victims of assault and had taken personal offence at the Prime Minister's accusation.
Despite a warning from the Speaker, MPs continued to stand to urge him to take action.
Green MP Marama Davidson was then thrown out of the House, followed by Poto Williams.
They were followed by at least eight others who chose to leave the debating chamber.
Speaking outside the Chamber after the walkout, Green MP Jan Logie said it was the first time she had spoken out loud about her own abuse and it was not easy.
"It's deeply disappointing that the Speaker and the Prime Minister do not take the concerns of sexual survivors seriously. It's completely unacceptable to trivialise the concern and experiences of so many New Zealanders in the way that has happened today."
Labour MP Poto Williams said the Prime Minister often wore the white ribbon that was a symbol against domestic violence. "He now has to live up to that pledge. The men across the House who wear the white ribbon who don't actually believe in the tenets it promotes that they don't condone violence, they must withdraw those ribbons immediately."
She said the Speaker was bound by the rules and it was up to the Prime Minister to apologise of his own accord.
"It's not an easy thing to do to make those disclosures in a very public way, so we are calling on the Government and the Speaker to urge the Prime Minister to do the right thing."
Labour MP Clare Curran said the Speaker's decision to evict some of the women who had raised their own experiences sends a bad message.
"The Speaker today threw out women from the House of Representatives for standing up and saying they were the victims of sexual violence. What message does that send to young women?"
Earlier, Deputy Speaker Chester Borrows said matters of privilege had been raised regarding comments Labour leader Andrew Little and Chris Hipkins had made about the Speaker.
Mr Borrows referred to two reported statements. In one, Mr Little said the Speaker had "a reputation ... of acting politically in the way he handles question time".
Mr Hipkins had said "the Speaker is interfering on their behalf to make sure the Government does not face that embarrassment".
Mr Borrows said the matters of privilege were significant because they accused bias against the Speaker.
"Allegations that the Speaker acts on the instruction of the Government and interferes in Parliamentary process for political reasons are serious matters that suggest the Speaker is not only biased at performing his duties but is open to partisan manipulation."
The comments were pre-meditated and had not been withdrawn, he said.
As a result, a question of privilege had arisen in both Mr Little's and Mr Hipkins' comments and he had referred the matter to the privileges committee.
Once he had taken the chair, Mr Carter said he wanted to remind ministers to refrain from making irrelevant or abusive comments as part of their answers.
He also addressed Labour MP Kelvin Davis' confrontation of Mr Key outside the House yesterday.
He said robust debate should only occur in the debating chamber and in select committee rooms.
"For such robust debate ... to be taken outside the chamber ... steps into an area of danger," he said.
"I would hope members would consider this carefully in future."
During Parliamentary question time, Mr Key did not back down on his comments but gave further detail about offences committed by New Zealand detainees on Christmas Island.
They included indecent assaults and assaults of children, police officers, and others.
Mr Little pointed out that 130 of the total New Zealanders waiting to be deported had convictions for low-end offences such as cannabis supply and shoplifting.
He said Mr Key's attempts to portray them all as rapists and murderers was "disgraceful".
