"I got home and called Spector's office about midnight and said, 'I'm not going to be able to make the rest of the week'. It wasn't fun."
Carlton's career in studios saw him contribute the guitar solo to Steely Dan's Kid Charlemagne (considered by Rolling Stone magazine to be one of the three best in rock), record with everyone from Sammy Davis jnr to Dolly Parton, play on dozens of soundtracks and television themes (that's him on the Grammy-winning theme to Hill Street Blues) and also run a parallel career as a producer and player with the jazz group the Crusaders.
Since the late 70s he's had a high-profile jazz career, but recalls his first paying studio job which launched him into more than 3000 sessions.
"I was about 18 and there was a local producer, the drummer in the surf band the Challengers. He was going to do an album of cover instrumentals and called me because I could read music well but still sounded young.
"So I went in and there were Hal Blaine, Joe Osborn, Larry Knechtel...," he says, naming key players in the formidable Wrecking Crew who anonymously played on songs by the Beach Boys, Monkees and dozens of others. "I don't think I knew who they were and when I came out and told people I was surprised [when I found out who they were]."
Because he was a quick study ("Yes, I was," he says almost apologetically) he kept being called back. People like Mitchell and Steely Dan were easy to work with because they would just leave space for him to fill.
"We didn't work very hard on Joni's records, it was all very natural. But there were dates when the artist wasn't there. When I recorded She's Out of My Life for Michael Jackson, he wasn't there. I was honoured though, because I wasn't doing session work at the time but [producer] Quincy Jones wanted me on it."
Curiously, given the years he put in, session work didn't initially appeal: "I'd seen those session guys when [I was] doing that surf music album. They were pale, cigarettes hanging out of their mouths and looked really beat-up. I left thinking I didn't ever want to do that. I just wanted to play jazz in a smoky nightclub.
"But back in the day if you were part of what we'll call the A team [as a session player] you could make a very nice living and not travel."
His solo jazz career began in the mid-70s when playing Tuesday nights at Dante's, a jazz club in North Hollywood, with Jeff Porcaro (session drummer, later of Toto), Joe Sample and Pops Popwell (both Crusaders).
One night an executive from a record company was there and afterwards offered him a contract.
"At that point I'd quit session work somewhat and was stretching out. Because I was playing with enthusiasm again I just followed along. It was a great ride."
And still is. At 66 he tours as much as he wants, is in demand for workshops and helms his own jazz quartet. (The group with him here however are Australians hand-picked by his Gold Coast-based friend Louie Shelton, a former LA session guitarist who produced Seals and Crofts).
Larry Carlton is an acclaimed jazz guitarist and was a session man without parallel when he chose to be. But he wears each accolade lightly.
"Because of when I was born I got to digest each style - rock 'n' roll, surf music, rock and jazz - as it came up. So if someone wanted a certain thing I was familiar with it."
He's also a survivor. In 1988 he was wounded in throat in a random shooting, five years after he became a Christian.
"A number of Christian musicians, when they found out I'd had [a religious] experience and was committed, asked if I was going to start playing Christian songs.
"I said, 'No! God hasn't told me to do anything like that'," he laughs. "I was still allowed to play jazz."
Who: Larry Carlton Quartet
Where: Bruce Mason Centre, June 6; Opera House, Wellington, June 7
- TimeOut