He then added: "And I always said, 'Well, who hasn't?' to myself. But then, when I was able to see from the epicentre, the ripples, I thought, yeah, they could have fired me. But they didn't. So I owe them that."
CBS producer Robert Halderman was ultimately charged with first-degree attempted grand larceny after threatening to reveal Letterman's dalliances with this former intern, Stephanie Birkitt, if he did not pay him $2 million.
Birkitt began working on the show when she was 21-years-old and still attending college at Wake Forest. Letterman was 49 at that time.
Bikritt later became a member of the CBS page program, and eventually became Letterman's assistant after graduating college.
The two developed a close relationship that ultimately led to her making on-air appearances starting in 2002.
The scandal broke in 2009 shortly after Birkitt ended her relationship with Halderman, who she had been dating, and moved out of his home.
It was alleged she had continued her relationship with Letterman while with Halderman, and that the producer had used her emails and diary entries to blackmail the host.
Letterman was also in a relationship at the time with Regina Lasko, who he had wed just months before after years together, and who he had a son with, Harry.
Letterman revealed the details of the extortion on his program, admitted to his affair and said he was working with the district attorney to apprehend whoever was responsible.
When asked why he was so upfront about the incident, Letterman said: "I didn't know what else to do. I couldn't think of a really good lie."
In the aftermath, another former intern, Holly Hester, would come forward to say she too had a relationship with Letterman while working as an intern on the show in the 90s.
As for Stephen Colbert, the man set to step into Letterman's shoes, the 68-year-old says no one asked for his opinion about who should be brought in, and that he thought it would be another Comedy Central host taking over.
"I always thought Jon Stewart would have been a good choice. And then Stephen," said Letterman.
"And then I thought, well, maybe this will be a good opportunity to put a black person on, and it would be a good opportunity to put a woman on. Because there are certainly a lot of very funny women that have television shows everywhere. So that would have made sense to me as well."
Some had thought Chelsea Handler, who had found massive success hosting Chelsea Lately on E!, might replace Letterman, but she ended up signing with Netflix to do a new show.
And Letterman admits he was a little hurt that no one came to him for input.
"Just as a courtesy, maybe somebody would say: 'You know, we're kicking around some names. Do you have any thoughts here?'" he said.
"But it doesn't bother me now."
- Daily Mail