"It was crazy, I was in limbo with my visa, and yet here I was walking through the back door of the White House.
"I thought, if I bump into Obama, what do I say? 'Hey, can you hook me up?"'
Hoy Fong, who now lives in New York with his wife, was born and grew up in Epsom.
Despite a long-held ambition to be a chef, he studied business and science at Auckland University.
"You know - traditional Chinese family, Mum and Dad wanted me to go to uni and get my degrees and all of that," he explains.
After working in programming for IBM in Wellington for two years, he moved back to Auckland to help set up and run the family business, Fruit World Silverdale.
When a buyer came along after three years, Hoy Fong found himself with a bit of money and no commitments.
After an intensive six-month course at the French Culinary Institute in New York he did some obligatory hard-yards, for free, in top restaurant Daniel and Peter Gordon's Public, in Soho, London.
"It was a lot of hoping and working hard," Hoy Fong recalls, smiling.
With little money and an expired working visa, his fortune changed when he met Food Network celebrity chef Tyler Florence, who has been described as the States' answer to Jamie Oliver.
After working for Florence for free, Hoy Fong was hired as his sous chef, which enabled him to stay in the country.
His role soon changed to culinary director - responsible for everything from helping develop Florence's recipes and helping him manage his restaurants, TV show and other appearances.
That has taken him to the set of Oprah, and an appearance with Florence on the long-running cult show Iron Chef.
Now, Hoy Fong has started his own business and is working on his brand.
A year has been spent developing Top Chef University, an online culinary school based on the Top Chef show, the US version of Master Chef.
He is also getting used to playing the critical role of judge on TV shows, and this year he praised the "big, bold flavours" of rapper Coolio's cooking.
"Being a New Zealander, you don't really put yourself out there and say bad things. You just keep it to yourself, eh. It took a while to get used to," he says.
"I think I'm in a place now where I have got a point of view to share with people, and tell them what my take on food is."