Mr Key shot to his feet and stood until the incident was over. After sitting down, he said to Labour that they should be ashamed.
Labour deputy leader Annette King and at least two other MPs reacted angrily, shouting "scumbag" at him, and leader Phil Goff said it was a "sad incident" and even sadder that Mr Key had tried to blame Labour for it, at which Mr Key rolled his eyes.
Afterwards, Mr Goff said he was disgusted Mr Key would try to blame Labour when the man clearly had mental problems and was in a potentially life-threatening position.
He believed Mr Key was claiming it was a set-up by Labour.
"It was totally inappropriate behaviour for the Prime Minister of New Zealand.
"It's a disgrace he would even make that allegation."
Mr Key's office said soon after that the Labour Party's interpretation was incorrect.
"The Prime Minister's comments following the incident were a reflection on the Labour Party's continued criticism of his ... security detail - which included criticism of his being accompanied inside the parliamentary complex."
A spokeswoman for Mr Goff said Mr Key's justification for his comments was nonsense.
She said several Labour MPs had heard Mr Key's comments and it was clear he was blaming their party for the man's actions, not making a reference to security issues.
Unlike many overseas Parliaments, there is no barrier other than a low balustrade between the public gallery and the debating chamber below.
During the incident, several MPs rose to their feet and Ms King shouted "Move! Move!" to the MPs sitting directly beneath the man - Su'a William Sio, Rajen Prasad, Ashraf Choudhary and Kris Faafoi.
A security guard managed to reach the man just as he climbed on to the top of the balustrade to jump off.
The man continued to struggle over and at one stage most of his body was dangling over the seats below before guards and members of the public hauled him back over.
The first guard was later treated for cracked ribs and a broken ankle.