"But every time I sing they always said to me, 'You have a good voice why don't you sing? Why don't you want to be a singer? And I did know I had a voice. But ... "
But, well, it took a stint at Studio One with Dodd for the teenager to realise his true musical calling was singing.
"When I was in the studio and I started hearing my voice I couldn't believe it," he remembers. "When I first went to Studio One I thought I could sing. But I realise I had to start all over again. I had to learn all over again. Studio One was elementary, high school, and university all in one."
And ever since then, singing and reggae music has been his life. "It is living. Reggae music means life to me. It's the best music in the world, man."
In a career of songs that have touched on everything from intense and fiery political and social commentary through to sweet lovers' rock, Andy agrees Skylarking is probably his most enduring and possibly best-known track.
"It's true man. That song does go on and on and on. It means a lot to me, man, because I used to get up to lots of things when I was young." He laughs referring to his wayward youth in Kingston, which he sings about in the song.
He went on to record with producer Bunny Lee in the mid-70s (who, among others, worked with dub pioneer King Tubby) during a prolific period during which time he released You Are My Angel in 1973. Since then he has released many solo and collaborative albums - and hit a new height of popularity after joining forces with Massive Attack in 1991.
Andy returns to New Zealand this month with his band Dub Asante to support Shapeshifter on their summer tour - and though he admits he's not too familiar with the Kiwi band's songs he likes what he has heard. "All I know is they are great musicians."
He was last here for Womad in March and before that supporting and guesting with Massive Attack in 2010 off the back of the band's latest album (and return to form) Heligoland with Andy featuring on the agitating beauty Girl I Love You.
It was his collaboration with the Bristol trip-hoppers on 1991 debut Blue Lines, most notably on One Love, that exposed him and his music to a whole new - and younger - generation.
"The young ones come out and listen to me these days," he says sounding chuffed, "and that is thanks to Massive Attack."
He remembers fondly how he first hooked up with Massive Attack in London in the early 90s when the sound of trip-hop was about to take off.
"I was going home to Peckham, and I hear there was a group and they want a singer, and it turned out it was me," he says. "It turns out I was [Massive Attack's] Daddy G's favourite reggae singer. That what make it so nice, man.
"There is respect there man.
"They give me a lot of respect and I give them respect back. And as an artist I'm looking to change all the time."
That's why he has worked on all five of Massive Attack's albums. You can tell he's particularly proud of Angel - the haunting and heavy reworking of his chinking and jaunty reggae hit - as he breaks into that crooning mellow lilt of his to sing a few bars of the song down the phone.
LOWDOWN
Who: Horace Andy
What: Reggae singing great
Where and when: Ascension Estate, Matakana, Dec 29; Riwaka Hotel, Jan 2; Brewers Field, Mt Maunganui, Jan 6; Waihi Beach Hotel, Jan 7
Essential listening: Skylarking (1972). With Massive Attack, Blue Lines (1991) and Mezzanine (1998)
-TimeOut