The then 44-year-old had apparently caused a scene when found wielding a sword around the pub.
He later told media he was just carrying the sword on his shoulder and was "showing off his craftsmanship" as he had made the sword himself.
"Someone tried to take it off me and I didn't enjoy that very much."
Mr McPhee said the incident was not a big deal and no one had been hurt.
In 2004, Mr McPhee tipped up sitting mayor Martin Tankersley and held the mayoralty until relinquishing it in 2010.
He is remembered as a colourful but effective mayor who was staunch in his support for his town.
During his mayoralty, Mr McPhee also had a brush with the police after an incident in High St Carterton after a fracas in a flat in which a door was damaged. A confrontation with others led to police being called.
Later Mr McPhee said he had "rightly or wrongly intervened" when visiting the premises, forcing open a door and confronting those inside.
No charges were and no official complaints were made.
Among the first to pay tribute to Mr McPhee was New Zealand First Deputy Leader Ron Mark, who was sought out by Mr McPhee to take over the Carterton mayoralty in 2010.
"And he wasn't taking no for an answer," Mr Mark said.
"I always felt honored and humbled by the fact that this guy, who made such a huge impact on Carterton chose me as someone he felt could succeed him.
"I respected the hell out of him. He was larger than life. He's a big man, big in stature, big in heart, big on people, big on getting stuff done, and genuine, absolutely genuine."
South Wairarapa Mayor Adrienne Staples said she first met Mr McPhee at New Mayor's School more than 10 years ago, where he "absolutely stood out because he was a foot taller than everyone else".
"Gary and I were elected to our respective mayoralties at the same time and right from the start I would say Gary was delightfully a square peg in a round hole," she said.
"He did things differently and was somebody very special."
Mrs Staples remembers the last time she saw Gary, at the combined council meeting at the start of May.
"He would always give me a big hug, and he seemed to be down.
"I said how are you going Gary? And he said, 'okay'. I said that doesn't sound like you, can I help? He said no. I noticed then that he was really down and I know he hasn't been well lately. So we were concerned about him because he just wasn't himself."
Mr McPhee had gone on record disclosing his battles with depression.
Carterton Mayor John Booth described Mr McPhee as the "Sheriff of Carterton" - "a man mountain with a heart of gold".
"He was a hands-on person in the community. I had a lot of time for him.
"When he was the Mayor, he would visit businesses with a clipboard and pen and go into the shops asking, what do I need to fix?- He was that sort of mayor, connected and deeply committed to the Carterton community.
"He was colourful, and a real man's man.
"I can recall that when I became mayor in 2014 he rang me up and said he was very proud to have another Gladstone man as mayor of Carterton. That is one of my fondest memories.
"We crossed paths several times at the Gladdie Pub where he would arrive on his motorbike and occasionally ask me, what are you drinking? I'll miss him terribly."
Masterton Mayor Lyn Patterson said she was "devastated" by the loss of Mr McPhee.
"It'll be a huge blow for his family and the Wairarapa community.
"He was a big man with a big heart and was part of our Governance Review Team for the past four or five years."
Although Mrs Patterson had no mayoral interaction with Mr McPhee, she was familiar with him through her work at Carterton District Council prior to becoming Masterton mayor.
