The then 44-year-old had apparently caused a scene when found wielding a sword around the pub. He later told media he was just carrying the sword on his shoulder and was "showing off his craftsmanship" as he had made the sword himself.
Mr McPhee had said the incident was not a big deal and no one had been hurt.
In 2004, he tipped out sitting mayor Martin Tankersley and held the mayoralty until relinquishing it in 2010.
He is remembered as a colourful but effective mayor who was staunch in his support of his town, and not afraid to speak his mind.
During his mayoralty, Mr McPhee had a brush with the police after an incident in High St, Carterton, where he is said to have been involved in a fracas in a flat in which a door was damaged. A confrontation with others had resulted in police being called.
Later Mr McPhee said he had "rightly or wrongly intervened" when visiting the premises, forcing open a door and confronting those inside. No charges had been laid and no official complaints were made.
Among the first to pay tribute to Mr McPhee yesterday was New Zealand First deputy leader Ron Mark, who was sought out by Mr McPhee to take over the Carterton mayoralty in 2010.
"And he wasn't taking no for an answer," Mr Mark said.
"I always felt honoured and humbled by the fact that this Loss a shock to community
guy, who made such a huge impact on Carterton, chose me as someone he felt could succeed him.
"I respected the hell out of him. He was larger than life. He's a big man, big in stature, big in heart, big on people, big on getting stuff done, and genuine, absolutely genuine."
Mr McPhee opened the Buckhorn Bar and Grill in Carterton, and Mr Mark said the western-themed restaurant will "always stand as a testimony of his artisan skills, his artistic skills".
South Wairarapa Mayor Adrienne Staples said she first met Mr McPhee at New Mayor's School more than 10 years ago, where he "absolutely stood out because he was a foot taller than everyone else".
"Gary and I were elected to our respective mayoralties at the same time and right from the start I would say Gary was delightfully a square peg in a round hole," she said. "He did things differently and was somebody very special."
Mr McPhee had gone on record disclosing his battles with depression.
Carterton Mayor John Booth described Mr McPhee as the "Sheriff of Carterton" -- "a man mountain with a heart of gold".
"He was a hands-on person in the community. I had a lot of time for him.
"When he was the mayor, he would visit businesses with a clipboard and pen and go into the shops asking, what do I need to fix? -- He was that sort of mayor, connected and deeply committed to the Carterton community.
"He was colourful, and a real man's man. I can recall that when I became mayor in 2014 he rang me up and said he was very proud to have another Gladstone man as mayor of Carterton. That is one of my fondest memories. We crossed paths several times at the Gladdie Pub where he would arrive on his motorbike and occasionally ask me, what are you drinking? I'll miss him terribly."
Masterton Mayor Lyn Patterson said she was "devastated" by the loss of Mr McPhee.
"It'll be a huge blow for his family and the Wairarapa community. He was a big man with a big heart and was part of our Governance Review Team for the past four or five years."
Greater Wellington Regional Council chairman Chris Laidlaw could not be reached for comment.
