The family built a shed/garage that was to become the cellar door. "We lived in that for three years with the kids. It was very cosy," says Millar.
He and his wife have five children between the ages of 15 and 27 and the two younger children came with them. The couple have since built a house.
Millar asked neighbour, Rob Cameron, the winemaker and director of Invivo Wines, and the man behind the Lochiel Estate Vineyard and Winery, to do his winemaking.
Around 10 or 12 commercial vineyards are operating in the vicinity, he says.
The vineyard's first vintage came out in 2007 after planting in 2005.
Around 600 cases of wine are produced a year, and Millars wines have won three bronze medals in the University of Auckland Upper North Island Wine Challenge for its viognier 2010, gewurztraminer 2011 and syrah 2010.
Millars also won silver in the New Zealand International Wine Show for its Rose 2010.
More is to come with the redevelopment of the olive grove. "We may put chardonnay grapes there. Commercially, chardonnay is a good wine to have in your cellar," Millar says.
He has had a band rotunda built for concerts and weddings, and to provide a setting for locals and tourists to come and enjoy the wine.
There is also a picnic area and petanque court.
Millars Vineyard does up to 20 weddings in a season.
In another part of the property, Millar is thinking of moving an old chapel there. He'd also like a bell tower.
"People ask about the restaurant side, but that's a big step," he says.
Jennimay has a key role at Millars.
"She's very clever and switched on and she's got a good heart. She's enormously helpful as a front office person," Millar says.
The swim school has financed the setting up of Millars Vineyard so far, but Millar would like to get a small profit from the new business.
"It's amazing what you can pull off with a small vineyard," he says.
Meanwhile, the Ross Millar Swim School runs all but three weeks a year with 10 staff from Three Kings Primary School.
"The swim school teaches 3500 students a year, and we have taught for 21 years," Millar says.
"As an industry for us, it's been wonderful. I love going to work,"
Fortunately for Millar, his oldest son, Jordan, 25, is an excellent swim instructor and he is expected to take over the running of the business after his OE.
"It's comforting that the business will carry on."