Most of all, though, there is the palpable sense of frustration, the unfairness Armstrong feels at being cast as cycling's devil incarnate, its pariah, when others, who effectively made the same decisions to dope that he did, have already served their bans; some of them have even welcomed back into the sport with open arms. "I'm that guy everybody wants to pretend never lived," he says. "But it happened, everything happened. We know what happened. Now it's swung so far the other way... who's that character in Harry Potter they can't talk about? Voldemort? It's like that on every level. If you watch the Tour on American TV, if you read about it, it's as if you can't mention him."
Armstrong shakes his head. "It will not be the case forever because it can't be the case forever. That won't work, people aren't stupid."
We are sitting at a table outside Armstrong's holiday home. It is the end of a training camp for many of the riders taking part in next month's One Day Ahead, the charity ride set up by former England footballer Geoff Thomas, which is aiming to raise £1?million for Cure Leukaemia.
For two days Armstrong has led the group on local routes around Aspen; up a dirt track on the back of Snowmass, one of the biggest ski areas in the region, where Armstrong points out a mansion belonging to Chelsea's oligarch owner Roman Abramovich (a former riding partner); out to Ruedi Reservoir, straddling the county line between Pitkin County and Eagle County. A handful of UK journalists have been invited along by Thomas, to raise awareness for the charity.
On the way back we stop for lunch at Woody Creek Tavern, a favourite haunt of celebrated gonzo journalist Hunter S Thompson. Armstrong insists on reading out Thompson's famous daily regimen. "Ten [am] drops acid; 11, Chartreuse, cocaine, grass; 11.30, cocaine, etc, etc; 12 midnight, Hunter S. Thompson is ready to write." Armstrong will later insist that "none of this was an act". It is all faintly surreal.
He is easy company; the famous charisma of old clearly still there. He jokes with his new team-mates, adopting a Dick van Dyke-style British accent, teasing them about their riding, praising them when they have done well, racing them to the top of climbs. Although mainly he is happy to sit back and ride at a slower pace.
Armstrong says he hardly ever rides on the road any more - and won't touch his bike again before riding on the Tour route next month. He does not watch cycling on TV, does not follow it on Twitter or the internet. He has "no idea" who will win the Tour. Has he fallen out of love with cycling? "More or less," the American replies.
On the middle evening, Armstrong and his partner Anna Hansen host a BBQ for the British contingent at their home to which he invites other family and friends. His children - three by his ex-wife Kristin and two by his current partner - mill around the house, running in and out of rooms.
Beneath the bonhomie, though, it is not difficult to discern an undercurrent of suspicion and general unhappiness.
Armstrong looks tired; every one of his 43 years. I am reminded of an interview I did last year with Christophe Bassons - cycling's so?called Mr Clean who was drummed out of the sport during Lance's heyday for refusing to kowtow to the dopers - in which the Frenchman spoke about Armstrong flying over in the wake of his ban to apologise to him personally. Armstrong drank so much, and looked so depressed, Bassons said, he went away feeling that he was a suicide risk. "Maybe Lance Armstrong is stronger than I imagine but I don't want to hear that he has been found hanging from a ceiling, because I think it is possible," Bassons said.
I ask Armstrong what he felt about Bassons's comments. He shakes his head. "I mean, any answer I have for that, any true answer, is just going to look like the most arrogant shit**** answer," he says. "You know, it's funny. When I showed up there, I had been travelling. I hadn't shaved in, like, months. And I walked in and they had all this, they had L'Equipe there, and TV, and there was all these people... I was like 'woah, woah...what's all this? I'm not doing that'. So we settled on just talking, face to face, with a couple of journalists, and it was fine. I appreciate his concern but that [suicide] is not a problem. It's fine."
He remains, though, a self-confessed pariah, living a sort of luxury purgatory between his homes in Austin and Aspen, venturing out to the golf course most days - "Now that might lead to something dark. That f------ game! It doesn't get better" - or to the surrounding trails. He does not go to shops or bars or nightclubs "because I don't want to talk to strangers, sit around, make small talk".
He prefers to surround himself with trusted friends or family. Scott Mercier, the former team-mate who quit the sport because he did not want to take drugs, is now one of Armstrong's best friends. He joins the group for part of the Saturday ride. Todd duBoef, the president of Top Rank Boxing and one of the most powerful men in boxing, who recently promoted the Floyd Mayweather v Manny Pacquiao superfight, is staying in Aspen for the weekend and comes over.
It is a surreal glimpse into the strange world of Lance Armstrong three years on. A life in limbo. Not that self-pity will win him any sympathy. He is not exactly living on the breadline.
At 2,400m, the air in Aspen is almost as rarefied as the company. The famous ski resort, nestled in a remote area of the Rocky Mountains, boasts the most expensive real estate in the United States. Abramovich is only seventh on its rich list. Stanley Kroenke, Arsenal's majority shareholder and owner of the nearby Denver Nuggets basketball team, is way down in 15th place.
Armstrong, despite the doping, the cheating, the intimidation, the threats made against anyone who stood in his path, the countless lawsuits, is still rubbing shoulders with some of the richest people on Earth. And this is not even his primary residence. For some he will always remain an inveterate cheat and a liar; the recent traffic incident in Aspen, just a few streets from where this interview is taking place, which saw Hansen plead guilty to careless driving to save Armstrong negative headlines yet another example of a man unwilling or unable to take responsibility for his actions. For many it is the apparent lack of contrition that grates. Armstrong says he is "evolving as a man on that".
He and Hansen both see a counsellor regularly. "We can all be better people," he says. "God knows I could. I mean, I was a complete dick for a long time. I led a life that for 20-30 years everybody just stood around and said 'yeah' 'yeah' 'yeah' and then there was another 'yeah' and then a whole bunch more 'yeahs'.
"That leads to a real complicated... especially at a young age, from early teens, and then you add in success and victories and money and fame and momentum. That is no way to learn how to handle personal interactions."
"But I'm not going to be sorry for certain things," he adds. "I'm going to be sorry for that person who was a believer, who was a fan, who supported me, who defended me, and ended up looking like a fool. I need to really be contrite and sorry about that. And I am. But these other stories [CIRC, the whistleblower case etc] are so high profile, so hot. It's almost like a side business. That's too much. I'm more worried about Mary-Jane in Ohio, and Doug in Pennsylvania, or Liam in Birmingham or wherever. Those are the people who need... listen, if you could walk the world and face to face apologise I would. And all the others who were directly impacted, whether it be [Filippo] Simeoni, or Bassons, or Emma [O'Reilly]. I mean, I tried to make it right with every one of those people. I can only do so much."
There will be many who would like to see him stripped of everything he has before they are prepared to consider forgiveness. That may still happen. Armstrong has a federal court case pending which he claims could ruin him financially should he lose. "Yeah, I mean, the whistleblower case is a $100? million case," he says. "We would not be sitting at this table any more. We wouldn't be sitting in this home any more. We wouldn't be sitting in any home. I don't have $100?million."
Armstrong, though, admits that he "likes" his case. Not only does a federal jury require a unanimous verdict to convict - which he feels should play in his favour - but he is adamant that US Postal will not be able to prove it sustained damages. "In a whistleblower case you have to show... real tangible harm [to the federal government]," he says. "The Postal Service commissioned studies in 2004 that showed it made $100m. There were years when it was making upwards of $20?million a year in new business, before we had even started to race.
So when you start to add all these things up, here is the question: 'Where are the damages?'?" Armstrong sounds less optimistic on the prospects of winning a reduction in his lifetime ban, something he describes as "highly unlikely". He continues to talk with Tygart but says he has nothing left to offer in exchange for a reduction. "If we don't know it by now then I missed it. Between 10 books and three movies..." More names? More dopers? "Like? I don't know. I go back to this adult conversation I think there needs to be. Just by picking out people? You could say, I don't know, some random person who was on Postal - is that achieving anything? I mean it's just one more person that just gets... Anyway, if the question gets asked I'm going to answer the question, whether it's to him or in the federal case, I'm going to answer. It's not like I'm going to protect anybody. F---! At this point?"
Not that Armstrong wants a return to cycling. "What I want, and I had hoped the recent CIRC report would achieve this - but what I hoped that would achieve was that it would almost resemble some sort of adult conversation where we all just go: 'All right. Stop. This is really what happened. And this is who was involved and this is the line we are going to draw in the sand and this is where we are going to move forward.' But that didn't happen.
"There was all this expectation, there was a ton of money spent on it. I came through on my end. I said I would be the first man in the door, I did it, went twice, answered every question I could. The thing comes out and it gets panned. So we haven't had that adult conversation, I don't think. That's what I want. I'm not eager to be a team manager or walk around the Tour and have everyone pat me on the back." Why, then, does the ban even matter? "Primarily for triathlon, and it matters because the world was told I was the biggest fraud in the history of sport and I don't think that's true."
What Armstrong craves more than anything, though, is the chance to work again for his foundation Livestrong. And to do that he needs to be welcomed back into the cancer community. And this is where Thomas has come in, clutching an olive branch. It is a massive opportunity for Armstrong. "I'm grateful, sure," he says. "It's not like Geoff all of a sudden said 'OK, it feels like we might be safe now.' Because obviously we weren't. He and everybody involved caught a ton of grief. We knew there would be criticism.
"I was talking to Mark [Armstrong's agent Mark Higgins]: 'Dude, any day, this motherf----r's calling and going, 'All right, it's too much, I'm tapping [quitting].' And every time I heard, 'Geoff wants to talk tomorrow,' I thought, 'OK, here it comes.' Most people don't have the guts to ride that out but Geoff... that's a courageous thing."
Others such as Cookson and Sir Dave Brailsford, the Team Sky general manager, are less impressed by Thomas's decision. Cookson, for instance, who spoke for many when he called Armstrong's involvement in and around the Tour, even for a charity, "completely inappropriate and disrespectful". He even suggested that Armstrong might be at risk from irate fans, particularly if he ventured to a mountain stage such as Alpe d'Huez. (Armstrong relishes confrontation but it appears even he is not up for running that particular gauntlet. "That one we will probably just skip - Alpe d'Huez is a volatile setting.") He says he will do two or three days in total, but will not divulge which ones they will be.
In the main, though, he is not for turning. "Not to disrespect them [Cookson and Brailsford]. But I don't care what they think. It's not about them. I mean, I don't know Brian Cookson. I've never met him, never really had a conversation with him. I don't know what his vision is for the sport. But I do know that me and Geoff riding in France for this cause is the least of his problems. If he is making public comments - and this is as strong as I'll go - he needs to be talking about other things because this sport is not in a good place for a variety of reasons."
Does he not feel cycling has moved on? "I absolutely don't think it's in a better place. [With the introduction of the] biological passport combined with Landis testing positive, that was that moment when this s*** is getting real. The big shift did not happen in 2012. You guys can decide if he [Cookson] has done a good job, if he's been tough on Astana [the Kazakh-funded team who have kept their WorldTour Licence despite five riders affiliated to the team testing positive last year], whether he's stuck with his mission statement. Plenty of people would argue he's laid down on a lot of things. Whether it's expedited TUEs [Therapeutic Use Exemptions], Astana - Cookson is not very good at taking people down."
Armstrong is even hopeful that the reaction to He-Who-Must-Not-Be-Named might even be positive. "I could be wrong, I've been wrong plenty in my life, but I've been to France since all this happened and if you walk into a cafe or a restaurant or walk down the street that [negativity] is not the reaction I get. God forbid the reaction is positive. What happens then?"
On being cycling's Voldemort
Who's that character in Harry Potter they can't talk about? Voldemort? It's like that on every level. If you watch the Tour de France on American TV, if you read about it, it's just as if you can't mention him
On Geoff Thomas
I'm grateful to him. It's not like Geoff all of a sudden said 'OK, it feels like we might be safe now.' Because obviously we weren't. He and everybody involved caught a ton of grief. We knew there would be criticism.
On the reaction in France
I could be wrong, I've been wrong plenty in my life, but I've been to France since all this happened and if you walk into a cafe or a restaurant or walk down the street that [negativity] is not the reaction I get. God forbid the reaction is positive? What happens then?
On reducing his lifetime ban
It matters primarily for triathlon and it matters because the world was told I was the biggest fraud in the history of sport and I don't think that's true. I know that's not true. So I'd love to go to test myself in other things - not cycling things but whether it's an Ironman or whether it's something else. That's why it matters.
On Cookson's criticism
I don't know Brian Cookson. But I do know that me and Geoff riding in France for this cause is the least of his problems. If he is making public comments - and this is as strong as I'll go - he needs to be talking about other things because this sport is not in a good place.
