So when Baxter set about writing new songs, he felt burdened by the pressure of expectation. "It's a strange feeling when you know people are going to listen to your work." He found it impossible to write from that perspective. "A lot of it was me trying to figure out how to write from in here," he says, putting his hand on his chest.
"After that two-year period, people started forgetting about me and that was actually really nice and I started to feel normal again. I think that was when I started to be able to write like me again."
The album's first single, Inside Out, is a beautifully layered slow-burner that perhaps isn't quite the catchy earworm that Love Love Love was. But it's arguably better and its subtlety sets a warm and atmospheric vibe that's continued throughout.
Baxter describes it as a personal album, adding that some of that initial struggle in writing it was because he couldn't relate to the songs. However, those that made the final cut have meaning. And he's pretty damned pleased with what he's made.
"I just got the CD and I was listening to it on the way here in the car and I was like, 'This is a good album, I am happy with what I've done', which is rare as an artist," he says.
For a perfectionist such as Baxter, that's saying something.
We Are for the Wild Places is out now.