"With the Ross Shield happening now I've been thinking about that a bit."
Of horse racing and the colourful landscape it presented, he simply "grew to love it".
He would go with his family to the race meetings at Waipukurau and Woodville and still remembers the names of some of the horses which brought his nana a bit of luck now and then.
The 44-year-old also remembers 1982 and the horse he was sure would win the great Melbourne Cup, having already taken the Cox Plate for the third time - Kingston Town. "He came second."
But the Lindisfarne College old boy did not enter the racing industry workforce straight away.
"I worked for the breweries," he said, and agreed with a smile that yes, there was a touch of "rugby, racing and beer" to his life.
He worked for Lion Breweries in Wellington before returning to the Bay in the late '90s, where he took up a senior role with Carlton and United (Fosters).
He then touched the waters of racing by taking up a role at the Otaki-Maori Racing Club, before striking out for one of the country's premier courses - Ellerslie in Auckland.
He was Auckland Racing Club's race director for 11 years and the reputation for business, and his passion for the industry he took with him, soon expanded.
One of the Auckland club's CEOs, Cameron George, said Mr Castles was a hard man to replace.
"He has immense racing knowledge and management skills which made him an asset to this club," Mr George said.
They were attributes, he said, that had clearly been recognised by the Hawke's Bay Racing Club.
He took over the role previously held by Jason Fleming, who left to take on the chief executive's job at the Infinity Foundation.
While he and his family - wife Tess and youngsters 6-year-old Hunter and Francesca, 3 - were relishing being back in his home region, his first Bostok New Zealand Spring Carnival had been a tough road.
The first of the three scheduled meetings, the Makfi Challenge Stakes on August 29, went just fine - as the weather stayed fine.
But easterly storms scuttled the second event, the Windsor Park Plate, where $400,000 total stake money was up for grabs.
Stewards conferred and agreed the meeting should be abandoned after the third race as there was deemed to be a footing issue at the 800m mark.
"That was a devastating blow but it was out of our hands."
The weather prospects today, however, are a lot better, and the track has recovered remarkably well after the seemingly endless spring rain.
Mr Castles said a rain meter at the track showed 145ml of the wet stuff had fallen but the staff carrying out track work and inspections were smiling.
"The reality is, by Saturday the track will have effectively dried out - it's been quite amazing."
So things are, in the space of a drying week, pretty well set up; so what does he want for the big day which features the Livamol Classic?
"What we want to do is make sure we deliver on our promises to make it a great meeting," he said,
which means appealing to the public - across the board.
Families would be looked after, as would the kids, with activities set up to keep them happy and occupied.
"It's important to get everyone involved - to have things run smoothly and make them want to come back again."
Having now taken the reins, Mr Castles said there were challenges to be met, and top of the list were the facilities.
"People going out to events expect to have modern state-of-the-art facilities but we don't have that, so that is a challenge for us to work on."
He said there was also the challenge of engendering interest in racing, especially to younger people.
Across the country, racing was not in the same relatively high-flying state it was a decade or more ago, so therein lay the challenge of trying to spark interest - especially in the regions.
"Hawke's Bay is not a huge market."
In terms of stake money, that part of the racing game, in what are effectively more difficult times for the industry, was something of an unknown.
There was a time when the great Hawke's Bay Spring Carnival, with the running of the Kelt Capital Stakes, had million dollar marks beside it.
That, Mr Castles said, was at a time that was arguably at the outer edge.
Racing New Zealand governs the stakes although they could be added to by sponsors and businesses.
"We are able to increase them above their mark if we can provide more."
So it came down to backers.
"Times are pretty tough all right and it will be interesting to see where New Zealand goes - it's about getting it to the right size."
That may mean less racing and greater stakes, or it may be the other way around.
How it ultimately headed was up in the air, with the most important part of the equation being to provide punters, spectators, the general public, families - any potential race goers - with a great event.
"The people who come along on Saturday will have a great day and great time - but we don't want that to be only a couple of times in a year - we want to see them more than that."
Today, he will be doing his part in seeing as many people as he can.
"I'll be out and about getting an overview of everything."
That includes the events, the effectiveness of the marquee layouts, what people are doing.
And checking out the preparation and running of the 11 scheduled races.
"To make it a day people will enjoy and remember - and want to come back to."
