"What you don't want is to slow the economy down with something unnecessary, but on the other side of the coin that doesn't mean banks shouldn't be held to account for poor behavior. There's clearly been some in Australia and no one is saying everything is perfect in New Zealand."
He said the role of directors and executives in the financial markets is changing, citing the bank executive accountability regime recently introduced in Australia as one example, which sees bankers' pay deferred over longer periods and docked for poor outcomes.
Directors can also be struck off.
"It's a big deal and I think that's going to come in New Zealand," Key said. "Banks enjoy a certain position in society, they are very important for the economy but with that comes responsibility."
Key described the Reserve Bank and FMA review as a "deep dive" into this country's banking sector but even if there are no major issues there will be changes coming to the banking landscape on this side of the Tasman, although he did not advocate requiring Australian banks to list shares in their New Zealand subsidiaries separately on the NZX.
Meanwhile Key, who is also on the board of Air New Zealand, echoed his earlier warning in July about the threat of a global economic downturn.
While the US was delivering "Rock Star" growth, it's huge budget deficits following debt fueled stimulus programmes was of concern.
However, how China responds to US President Donald Trump's trade agenda was the biggest risk.
China's economy in October showed manufacturing activity continued to worsen, as the effects of a continuing trade war with the US hit home.
"China is a very complex big beast and it's a very different beast in macro numbers than it was nine years ago," Key said.
"And the challenge I think is that I don't think they know how this plays out with Donald Trump."
"Trump is not necessarily known as the most consistent leader in the world … but man he's consistently been opposed to trade. He's not going to let go.
And I think it's going to cause real consternation for the Chinese about what they do in response."
In terms of New Zealand's position, Key said the country was in a lot better shape than many other countries due to its popularity as a place to live, strong natural resources and commodities, relatively low government debt and an open economy.
However, there were a lot of potential headwinds.
"If ever there was a time to be cautious now would be that time."