"If I'm completely honest, my friend and I were more interested in backing the horses than mucking out the pens and so on. But I started doing some work out there.
"Anyway, as you do when you're 15 years of age, I came home one day to declare to my mother that I was leaving school and I was going to work for the horse trainer."
But Key's mother had firm views about the need for some financial security in his life. She made it very clear that he should forget about the horses and stick with plans to study accountancy at university.
"I can tell you that conversation lasted about as long as the KiwiSaver GST tax the other day," says the former prime minister.
Key seems to be in his element talking money - perhaps because he grew up without a lot of it, perhaps because he's since made a lot of it.
Whatever the reason, he is relaxed and happy to rattle through anecdotes and observations about personal finance and economics.
And Key isn't shy of sharing his best money advice or his most indulgent purchases.
"To be honest, I've just bought myself a helicopter because I've been learning to fly," he says. "It's only a little helicopter, it's a very cute little one and it comes next year."
That suggests that whatever his net worth, Key must be doing OK because he says the best financial advice he's been given is to live within your means.
He's not a fan of debt, he says, with the exception of using it to buy a property.
His other tip for young people: understand the power of compound interest and buy property.
"I know people will say that's a very hard thing to do if you don't have a deposit and you don't have the bank of mum and dad, I get all that stuff," he says.
"And it's not because I think its so tax advantageous, even though there are some tax advantages.
"It makes you save. You pay off some principal and some interest on your loan; if you have some spare cash or a bonus or something you pay off some more."
Now the chair of ANZ New Zealand (and on the ANZ Group board), Key is also on the board of global cybersecurity giant Palo Alto, an adviser to global media giant Comcast and sits on the advisory board of BP in London.
Locally he's also an adviser to Zespri and a board member of education consultants Crimson.
He got involved in Crimson after meeting founder Jamie Beaton through his son Max.
Key rates Beaton highly as an entrepreneur and sees lifting educational achievement as a key to improving New Zealand's performance.
It does all sound like a lot of work for someone who could afford to be doing much less.
"It's funny, my son said to me the other day 'why do you bother working? You could be playing golf or flying helicopters or whatever'.
"Partly it's intellectual stimulation and I enjoy it. Although I don't want to die working. I am happy to retire but I guess there is part of me that just feels I should keep just working to make sure."
Key famously grew up in a state house, and while he might not have been poor, money and financial security loomed very large - especially for his mother.
"There was never a situation where we couldn't eat," he says.
"Did I get Christmas presents? Yes, to the best of my memory. There was a lot of love and a lot of everything good. But we did grow up in a state house and did I go to other friends' houses and notice their things were cool and ours weren't? Truthfully, yeah."
Key's mother was so keen to instil the importance of money into her son that when he was about seven she took him to meet the bank manager at the local branch of the Canterbury Savings Bank.
"That was because she had borrowed money to buy a house and she said, 'look you have to know the bank manager'," he says.
"She was absolutely desperate for me to be an accountant and she thought: the bank manager - that's got to be the most important accountant in her mind that she knew.
"She had been an Austrian Jewish refugee - that got out when the Nazis invaded - and came from a wealthy family actually, to effectively no resources at all," Key says.
"She ended up in London pre-WWII with her and her brother not speaking English and with no financial resources. She had learnt that life can throw up these incredible things. Circumstances that you can never imagine.
"Money really matters - nothing terrifies people more than that they'll run out of money and have no money to retire on."
But money isn't the only driver.
If he was a slave to it, Key would never have dropped out of the finance sector to spend 15 years as a politician, he says.
"Initially working is about money and putting food on the table but the reality is there are so many other things that come out of that.
"It's that connectivity, it's going to be one of the really interesting things when one day we explore what the future of work looks like post-Covid.
"I reckon it will be a hybrid world, but I reckon that people will drift back."
Young people in particular learn by osmosis, he says.
"They learn by being there, they want to be noticed. And actually, working off the end of their bed in some grotty flat somewhere probably isn't that seductive."
Key eventually got his commerce degree and started life as an accountant in Christchurch - but he quickly knew it wasn't for him.
"He was quite young, thinking back. I was 21. I'd been in the job for three or four months and we were having a beer after work one night and I said to him: 'Do you really enjoy this?'"
His boss admitted the job was dull, he didn't enjoy it, and explained that it was just a means to fund a small farm in Templeton and other things he wanted to do.
Key recalls going home that night and saying to [future wife] Bronagh: "God I can't spend the next 40 years of my life doing something I don't want to do to pay the bills."
Key jumped ship, using a connection his sister had to land a job as a project manager at Lane Walker Rudkin - which at the time produced the Canterbury Authentics sportswear brand, and was one of the biggest businesses in Christchurch..
It provided a chance to hone his skills as a salesman, he says, recalling a big deal he did.
"One thing about men's fashion is we're attracted when see a shop window to all this gold and red and white ... and we don't buy any of it. We buy navy, grey and black.
"So anyway, these designers would design these incredible bloody ranges in all these colours," he says.
"We had one particular designer who went nuts on this and my boss said to me 'we have a warehouse full of this stuff, your job is to get this written off and get rid of it all'."
"So I went off to England and found this guy who was like an Arthur Daley type character and we sold the entire warehouse to him - tens of millions of dollars worth of it - and it all went off to Poland and Eastern Europe. So they were wearing the red shorts, they're probably still wearing it."
Perhaps Key could ended up as a men's fashion executive but, famously, he moved on.
He was swept up into the 1980s market boom as a currency trader working his way to the top at global investment bank Merrill Lynch before returning home to enter politics.
On reflection, he has just one regret from his time in politics, he says.
"I wanted to do a big cancer facility," he says. "I wanted to do the equivalent of Starship [children's hospital] for cancer.
"I got talked out of it. But if I had my time again I'd do it because I think generations of New Zealanders would look back and say that was a good use of our taxpayer dollars."
• Money Talks is a podcast run by the NZ Herald. It isn't about personal finance and isn't about economics - it's just well-known New Zealanders talking about money and sharing some stories about the impact it's had on their lives and how it has shaped them.