"He'd come into my shop when he was at Otago Boys High School; memorable, because he sounded off on anything.
"He had opinions on all the bands and music, but it wasn't nonsense. He knew what the right stuff was.
"He was very young, so consequently, when he started creating his own music obviously all the notes were in the right place."
Mr Colbert said Gutteridge was "ahead of his time", despite some difficult behaviour at gigs.
"He took a long time to tell you what the right notes were, in terms of tuning up at the start of [a gig].
"He was exasperating."
Gutteridge was in the United States this month for the first time.
"He's a man who didn't travel, and yet he's influencing bands in Europe," Mr Colbert said.
Despite drug issues that meant Gutteridge had played less for some time, both Mr Colbert and Port Chalmers venue operator Mike McLeod said that had changed recently.
Mr McLeod said Gutteridge had "cleaned up a bit in the last couple of years", and had had a productive period of playing, after a long time not playing live.
That included gigs in hometown Dunedin and dates in the North Island. "He was suddenly playing, and playing really well, and playing quite frequently," Mr McLeod said.
"I think he's an amazing musician."
A statement from the Flying Nun label described Gutteridge as "a great talent".
"All of us, and so many people around the world, have been touched and affected by his music, whether it be the swirling fuzz of the guitar or haunting piano melodies, Peter was a true hero of New Zealand music, and will be deeply missed."
Despite his role, the reclusive Gutteridge didn't enjoy being labelled part of the "Dunedin sound".
He told interviewer Wes Holland of messandnoise.com last year: "People didn't think about the sound of things, people put on guitars and then clanged out stuff.
"I just got tired of a guitar sound that wasn't thought about. I had my own personal style. I mean, I wrote [The Clean's] Point That Thing [Somewhere Else] at 17. That sort of sums up where I come from. I love textures. I love Indian music - now that's true psychedelic music without having to give itself a term.
"A lot of rock music leaves me cold. It's anal. It's self-indulgent. That's it. But there's great stuff too. Rock music is only rock music."
- Otago Daily Times, additional reporting nzherald.co.nz