"He absolutely loves the game," said one associate. "He's got keener in more recent times - he's got the bug.
"But he still doesn't play nearly as much as he would want to - he just doesn't have enough spare time."
In the past three years, however, the PM has managed to find more time for golf.
In 2012, Mr Key - a member of the Royal Auckland, Omaha Beach, and Wairakei International clubs - played in the NZ PGA Pro-Am seeding tournament at Jack's Point, near Queenstown.
He was paired with teenage sensation Lydia Ko and shot 21 over par - the same score as Hollywood actor Don Cheadle.
In January he was invited to play a round of golf with Mr Obama in Hawaii, where they were both holidaying. Mr Key was reportedly playing off a 15 handicap, while the most powerful man in the world was playing to 18 - a bogey a hole.
Mr Obama and Mr Key teamed up to play against Mr Key's now 19-year-old son Max and presidential aide Marvin Nicholson at the Kaneohe Klipper course on a military base 20km from Honolulu.
The leaders, perhaps unsurprisingly, claimed victory, and indeed did so "handsomely", Mr Key told the Kinloch Club after a round at the Jack Nicklaus signature golf course near Taupo recently.
In June, he told the Kinloch Club that he was playing off a handicap of 10.9, which "grossly overstates my ability".
Mr Key admitted he was the most nervous he had ever been on a course while playing in this year's New Zealand Open alongside Kiwi golf legend and winner of the 1963 Open Championship Bob Charles, and US star Rocco Mediate.
During his chat with Kinloch Club's staff in June, Mr Key talked about how he got into the sport.
"I started playing when I was about 15," he said. "One of my teachers at school was a keen golfer and he asked if anyone wanted to come along and take part in few clinics. He gave us a few lessons at the Russley Golf Club in Christchurch over the course of the year.
"After that I really only played socially and occasionally with clients for work. It has only been in the last three or four years that I have started to take my golf a bit more seriously."
He listed his most admired player as Tiger Woods; his favourite current player as US up-and-comer Jordan Spieth; his most loved tournament as the Masters; and the best course he's played as the Bel-Air Country Club in Los Angeles - "It's full of history and I shot an 81 there."
Labour leader Andrew Little, also a keen golfer, said Mr Key obviously had spare time on his hands.
Despite enjoying the game, Mr Little did not rate his own skills on the golf course - "I'm a hacker." additional reporting:
