"The permanent members have become used to exercising power and are protective of their privileged position. They presume to control the council's agenda and to determine its processes."
He said that made it difficult for the 10 elected members. There were also structural problems.
"The challenge for a small elected member to have its voice heard and its views taken into account is considerable. That the five permanent members of the council have the veto creates extraordinary power imbalance."
That was worsened by the practice of the permanent members pre-negotiating outcomes to the exclusion of the 10 elected members.
Mr Key said that was why New Zealand supported the proposal for the veto power to be restricted in cases of mass atrocities and genocides - a proposal that is also supported by France and the UK.
Mr Key said the failure to deal with Syria was evidence of the dysfunction on the council.
"It is time for the council to step up. It's time for members to set aside their vested interests and historical alliances in order to stop the violence and end the suffering."
While the council sat on its hands, thousands were dying and millions were displaced.
"We cannot afford to let the council go from an institution with failings to a failed institution. There must be a political solution to the conflict in Syria. And it must come soon."
Mr Key said the Iran nuclear weapons deal was a bright spot for the council, and should be the basis of something wider - "a fundamental reset of relationships".
"Nowhere is the re-setting of relationships needed more than in Syria.
"The question is not Assad or Isil (Islamic State). Both need to be dealt with."
Mr Key said the council could work well, such as implementing the Iran weapons deal. However s well as in Syria it had failed in Yemen, South Sudan and with the stalled Middle East peace process.
The speech followed yesterday's Security Council meeting of foreign ministers at which New Zealand representative Murray McCully slated the council for failing to reach any resolution on Syria.
Following that meeting, Russia's foreign minister and the US Secretary of State had met to discuss a "deconfliction" process after confirmation Russia had begun air strikes in Syria, giving the US only one hour's notice to ensure they were not in the way.
As he prepared to leave New York Mr Key said he was proud of the contribution New Zealand was making on the Security Council and the values that underpinned New Zealand.
"I also leave dispirited by the Security Council's failure to help not only the people of Syria but those in so many other countries."
Mr Key said a workable political solution in Syria would not be a perfect one. He also made it clear it was not just Russia he was criticising.
"Indeed, this is a classic case of the perfect being the enemy of the good. It will require international cooperation and compromise - on all sides."
Despite his fighting words, Mr Key was speaking to an almost empty assembly hall because many leaders have now left New York.
Amnesty International welcomed Mr Key's statement but said New Zealand must also bring practical solutions to the table.
The human rights organisation in particular welcomed the Prime Minister's acknowledgement the global refugee crisis and inaction by the UN Security Council, when faced with mass atrocities, were two of the key challenges of our time.
"The New Zealand government must ensure it doesn't simply pay lip service to protecting civilians in conflict and the global refugee crisis, but puts its words into action," said Grant Bayldon, executive director of Amnesty International in New Zealand.
