NZ First deputy leader Ron Mark then raised a point of order on why the party had been ruled against when it tried to table a transcript of the interview last Thursday.
Mr Carter rejected that, and warned Mr Mark against re-litigating the same point of order. Mr Peters did so and was ejected, upon which NZ First members walked out of the House.
Afterwards, Mr Peters said Ms Tolley had not corrected herself at the first available opportunity according to the rules, and the Speaker had allowed her to do so.
"All we expect here is a rather rudimentary understanding of judicial rules. You think we are going to take this sort of behaviour in this Parliament, then you would be 100 per cent wrong. And that's a signal we are sending to some others in that Parliament as well."
Mr Peters said he had no confidence in the Speaker but he would not say what further options he was considering.
Serco is in the midst of a rolling controversy regarding its running of Mt Eden prison and incidents of violence there.
From today, a Corrections Department management team has taken over the day to day running of the prison for the immediate future.
In June, Ms Tolley told Q+A that she would be open to private operators being contracted to provide more social services. Asked about Serco, she said "if they can deliver great results for people, why not?".
This morning Ms Tolley was asked if she still held the same view, given the Mt Eden prison controversy.
"I think with any of them...actually the quality of the contract is the really important thing.
So in the case of private providers we will want to see a bit extra. And with any provider we want to make sure there is good control there.
"I think I would be worried about any provider who was not performing well. And we've got a number of those - whether they are private or NGOs."
Serco, which has also been awarded a contract to run the new $270 million jail for 960 men at Wiri in South Auckland, is looking at running child services in the United Kingdom.
Labour's Kelvin Davis, who has been raising questions about Serco's prison operations, said those comments were "frightening" and showed the Government was committed to privatisation, regardless of evidence.