Every governor inherits their own unique challenges: when Wheeler took on the job in September 2012, Europe and the US were still deeply entangled in the fallout of the global financial crisis.
There were hopes that super low interest rates and central bank money printing had stabilised things and a recovery was about to get under way.
But that turned out to be false hope and a massive slump in oil prices took the rest of the commodity market with it - including dairy prices.
In New Zealand, the Reserve Bank hiked rates four times through the start of 2014 - moves it was forced to backtrack on.
Of the issues that critics have had with the bank under Wheeler's management, this is the most substantive.
It is also the one he is most comfortable addressing directly.
The whole world was caught by the oil price crash, Wheeler argues.
"At that time we had the economy growing strongly, the international economy was looking good at that point and our projections suggested that we would be moving to a positive output gap." In other words, it looked as though the New Zealand economy was about to hit the upper limit of its growth capacity.
"The view was shared by the private sector forecasters and the IMF, and market pricing built in a rise in the OCR [official cash rate]," he says. "So the expectation in the market was that interest rates would rise."
Then the oil slump kicked in.
"We saw a significant change in oil prices from US$110 to as low as US$28 ... and we saw a huge decline in whole milk powder and other prices. That forced us to re-evaluate."
Central bankers aren't commodity forecasters, Wheeler says. "I think the test is not whether you can forecast commodity prices, but whether you make appropriate decisions based on all the information you have at the time.
"And then, if the circumstances change on you, be open minded enough to be able to respond to that and look at the new set of data and the new set of circumstances that you are facing. I think the bank did that."
As it turned out, the local economy was robust enough to get through the dairy downturn without a recession. The Reserve Bank proceeded to cut rates through to November last year.
Since then, the OCR has been stuck firmly at 1.75 per cent, where forecasters expect it to stay for at least a year and possibly out as far 2020.
There is the potential for seven more cuts if we needed it to get to zero. There's petrol in the tank if we need to move.
Attempts to describe just how neutral the Reserve Bank's stance is right now have become almost a running gag with economists - ultra-neutral, anyone?
Whatever you call it, Wheeler departs with almost no debate about the need to shift rates. And he leaves his successor plenty of options to deal with any serious shift in economic conditions.
Was it the plan to leave things in neutral?
He laughs. "No. But it is good that there is the potential for seven more cuts if we needed it to get to zero. There's petrol in the tank if we need to move."
The 2014 hikes aside, is there anything else he would revisit about his time as governor?
"No, I don't think so," he says. "I think if you look at the overall picture, in terms of five years, the economy has performed very well over that time. We've had output growth of around 3 per cent annually, employment growth of 2.5 to 3 per cent annually.
"We've had headline inflation that has been pulled down by 5 years of negative tradables inflation. Headline has been around 1.5 and house price inflation has come down from 20 per cent to 1 per cent."
Financial stability and inflation are the two key areas on which Reserve Bank governors are judged.
While the Auckland housing boom made headlines, it was the rate of credit growth that was bothering the Reserve Bank.
When Kiwis borrow to buy houses, most of the funding comes from offshore, so as mortgage lending soars to cover higher house prices, the economy becomes more vulnerable to global financial shocks.
The imposition of loan-to-value ratios (LVRs) and the taming of the housing boom stand out as Wheeler's most obvious successes - in partnership with his deputy governor and head of financial stability Grant Spencer (who will succeed him for an interim six-month term after Wheeler steps down on September 26).
"We certainly wanted to see house price inflation slow," Wheeler says. "When you see imbalances develop in asset markets and you're in a low interest rate environment ... it's very difficult to raise interest rates substantially to fight an asset bubble.
"So we went into loan-to-value ratios with our eyes wide open because we knew we'd be getting into political territory and that's always uncomfortable for a central bank. Because housing is such an emotive issue for the country and it's the main form of savings for a lot of people."
That said, Wheeler seems unfazed by the political pressure coming to bear on the policy in the past few weeks, as house prices have stalled.
He didn't interpret PM Bill English's recent public comments about LVRs as particularly negative.
"Well, my sense was that the PM had a microphone put in front of him and was asked the question," he says of English's suggestion that the Reserve Bank ought to be considering when to ease the restrictions.
"I thought what he said from that point of view was fair enough and, yes, we do think about the conditions under which we'd relax them."
The relationship with both Finance Ministers - English and Steven Joyce - has always been very good, he says.
"It's always been very open and very frank. When required. They've always given me access whenever I wanted to see them, they've always made time, always listened. They'll express their views from time to time. Which is fine, just what you expect. But they do respect the authorities of the Reserve Bank under the Act."
We went into loan-to-value ratios with our eyes wide open because we knew we'd be getting into political territory ...
Wheeler says media stories about tension between the bank and Government are just wrong - a view backed by English himself.
"Some journalists have looked for conspiracies that aren't there," Wheeler says.
"That's been frustrating at times. Because it hasn't been the case." He has at times bitten back at those who he feels have overstepped the mark.
There was the rebuke to a senior BNZ economist - via an email to his boss - which went public after an official information request. Wheeler declines to comment on the incident.
And there was the Newshub leak of an OCR decision which prompted a ban from media lockups and ultimately a change in the way in which Monetary Policy Statements are released.
When I ask for the best and worst things about the job, frustration at the media coverage comes up.
Does he think he has been more combative than the past governors?
"I think central banks around the world - especially in the advanced economies - have been under quite intense scrutiny by the media and politicians," he says.
"We're in a world of unconventional policy - negative interest rates and quantitative easing - you're always going to have differences of view on monetary policy.
"Because it will affect the rate of employment growth and output growth and inflation expectations, it will affect the interest rates for savers and it can affect the exchange rate.
"So there are a lot of things that it can influence; you just expect that to be the subject of a lot of commentary and a lot of different views. I recognise everyone is perfectly entitled to their view."
He doesn't accept, though, that he is thin skinned. And while he doesn't want to revisit the specific stories, he makes it clear that where he has pushed back, it has been where he felt the integrity of the institution was being unfairly maligned.
Wheeler's World Bank track record should have offered advance warning that he wouldn't be afraid to stick his neck out. He famously called for World Bank president Paul Wolfowitz to resign at a meeting of all the senior team - with Wolfowitz in the room.
It could have been career ending.
Wolfowitz was under fire for code of conduct violations, but refusing to go.
The embattled president still had White House backing. After speaking out, Wheeler faced high level public attacks, including a series of Wall Street Journal editorials targeting him personally. In the end, he stuck it out and Wolfowitz quit.
It's safe to say nothing the New Zealand media has thrown at Wheeler has compared to the intensity of that experience.
OK, so what about the best things about the job? What has he enjoyed most about being governor?
"Working with colleagues," he says.
Wheeler has struck a close bond with his fellow governors - financial stability head Spencer, head of operations Geoff Bascand and head of economics John McDermott. With their support, he has moved OCR decisions to a committee based process.
He says he has enjoyed the management aspect of the job, being chief executive of an organisation he believes is full of world-class people.
"We've put a lot of time into trying to strengthen management through the institution," says Wheeler.
He has enjoyed thinking through the complexities of global economics and the interaction with politicians - on all sides of the political fence, he notes.
And, perhaps in contrast to his reputation with the media, he has loved communicating directly with the public.
"I've enjoyed the public speaking," he says. The bank does about 130 presentations around the country every year, 80-90 of them by the governor. "If you look at a map, we've got around all the small cities ... anywhere with a population of 20,000 or above."
So it's not all sitting in an ivory tower thinking, then?
"No, it's been hugely important. We get around to a lot of places that many institutions will never get to. So down the West Coast, Ashburton, Timaru, Invercargill, Whangarei ... That's been really enjoyable. I loved getting out and doing that."
Wheeler turns 66 next month, so he could be ready to retire. He isn't, though.
So what next? He laughs at suggestions he might need some time off, lying on a beach somewhere.
"No, there'll be plenty of time for that too, hopefully," he says.
Instead, he'll be picking up where he left off before becoming governor, on the board of Thyssen-Bornemisza Group - a high-powered European industrial and investment group.
"I've got a couple of board meetings coming up pretty quickly this year - one in Milan and one in New York," he says. "That will get me back to Europe three to four times a year and then I'll look for some board work in NZ and Australia and hopefully do some charity work."
On the local front, he hopes he can add value to companies with strong outward, global focus
And while he doesn't go into detail on charity plans, Wheeler talks with great passion about his experiences travelling and living on the breadline as a young man, and the work he did on poverty at the World Bank.
"One of the best things I ever did in life was, twice in my 20s, I took off travelling around the world. I did it deliberately to experience a different side of life and try and understand hardship, disadvantage and discrimination.
"You're essentially living on your wits, sleeping on streets, living with almost no money." It fed a passion to take his economic training to the World Bank, where he says he was lucky enough to get some breaks and do important work on poverty alleviation.
It's work that he clearly doesn't feel he has finished yet.
