"I had never been to the races but went to Ellerslie one day with $12 and went to town after with $84 from my winnings," says Ellis.
"I remember thinking, 'how great is this? This has got to be the greatest game in the world'."
Ellis was 18 then, with no family background in racing, a rarity for those who rise to the top in the sport.
"I am the only person in my family growing up who had any interest in horse racing or farming. Now we have Te Akau with 100 full-time staff, well over 100 horses and a farm with 2200 steers and 3000 lambs on it."
While his love of horse racing was born that afternoon at Ellerslie, Te Akau was born from a different racing experience a few years later.
"I had one of the first horses I owned win a race at Matamata and the trainer had to go saddle a horse up for the next race, so I went to the winners' bar to have a drink after and I thought, 'hell, this is a bit boring by myself'. It made me realise the real fun in racing would be winning with your mates and enjoying it together. So I started to buy horses and race them with friends."
That has grown into New Zealand's biggest horse racing business, and Te Akau's syndication model, powered by their enormous success, looks set to make them New Zealand's leading stable for potentially years to come.
Their young trainer Jamie Richards reached the top 10 in the world rankings this season and won three Group 1 races, racing's highest level, in Australia. With the stake money across the Tasman now stratospheric compared with New Zealand, Ellis will target Te Akau's best horses at the rich Australian carnivals.
"Karyn and I are proud New Zealanders, and one of the reasons I decided to buy better quality horses is because I hated seeing all the wonderful horses we breed in this country going offshore.
"And now we have so many very good horses, we definitely want to target the big Australian races more. We would love to fly the Kiwi flag over there and show them how good our horses are."
The stable has superstars Melody Belle, Te Akau Shark and Probabeel headlining their team, all Group 1 winners in Australia this season, and while New Zealand racing has its battles as it goes through a massive restructuring, Ellis says this is and always will be home.
"I don't feel like I have a job because horse racing is our passion, for both Karyn and I," says Ellis.
"Sure, we run a large business but it isn't a job when you are doing something you love. Just like back in that afternoon in 1971, I still love going to the races."
The couple are also joint patrons of the Waikato Riding for the Disabled.
"That is something very close to our hearts."
