And his experience in sales and marketing roles at clothing manufacturers meant he knew his plackets from his pockets.
There were two hurdles to overcome: getting a manufacturer to turn out the small quantities they needed and finding someone to run the business.
Miles was happily employed as general manager of sales at Bendon, a job he had no intention of leaving, but used his rag trade contacts on a fact-finding mission to Hong Kong.
He traipsed from one end of the island to the other, door-knocking potential manufacturers only to be laughed at when he told them he was after just 25 to 50 pieces made up in each colourway and design.
A glimmer of hope came when one high-end shirtmaker offered the possibility of using its sample room to make small runs using fabric left over from other brands.
With the manufacturing issue resolved, Miles realised finding someone to run a start-up retail business doing everything from store concept, product design and human resources through to financial management was going to be nigh on impossible.
"This thing, originally, was always going to be a little sideline. We could throw some money at it, find somebody to run it and we'd see where it goes."
Miles knew he'd die wondering if he didn't have a crack at it, so he chucked in his day job and opened the 3 Wise Men store near Broadway, Newmarket, in 2005.
The location was well considered - close enough to the Broadway shopping hub to get foot traffic, but avoiding steep Broadway rents. The strategy was repeated in Wellington, where the business opened its second store just off Lambton Quay.
"I remember day one we sold one pair of boxer shorts to a friend and I was standing behind the counter going 'that's great, I've left my job and we've just done $20 turnover for a sympathy buy from a friend'."
For a business with very little money to throw at marketing, friends became critical in getting the word out.
The three partners cobbled together a database of 800 people they knew, says Miles, hitting them with quirky emailed messages, the "three shirts for three hundy" price point and shirts with uniquely New Zealand names: the mint surf shirt, the shizzle shirt, the sticky tar day shirt, the booyah shirt.
"Through that network we were at least able to get them into the store once.
"The key was that we had great product that they would go back to work and people would say 'gosh that's a great shirt, where did you get that from?'."
Those guys talking about the little shirt shop in Newmarket at meetings and after-work drinks would became 3 Wise Men brand ambassadors, says Miles.
Complementing the retail store was an online offering, a rare thing for a menswear brand in New Zealand eight years ago.
Creating a clearance area to get rid of broken sizes and end-of-line stock was a big part of the online strategy, says Miles.
He was keen to ensure honest and consistent pricing, which meant no mark-ups to mark down, and no sales.
It also gave the brand a nationwide presence beyond its six retail stores for people wanting to repeat purchase.
The first of its two Sydney stores opened four years ago, a move Miles calls "hard yakka".
"Lots of lessons learned along the way ... I wouldn't say we want to scramble out of there but it has been much harder than we anticipated."
Miles says they totally underestimated the power of the Australian department store that could kit out customers from head to toe in brands they knew and loved, often at sharply discounted prices.
Add to that a parochial customer base loyal to Australian brands and eye-watering Sydney rents that pushed them into slightly secondary retail leases.
If Miles were to start from scratch he'd bring on board an Australian partner or retail brand with knowledge of the local market idiosyncrasies.
It's that strategy he'll use to take the brand to the home of shirting, Britain. Miles is nailing down a joint venture with a British retailer considered to be an online market expert.
In New Zealand volumes have grown from that first pair of undies sold eight years ago to about 2000 shirts a week and there is still plenty of scope for growth, says Miles, including a bold new retail concept.
And the lads still meet over a drink and "talk nonsense", he says, dissecting the business and planning world domination.
