7 Times David Letterman Thought He Should Leave The Late Show

On May 20, David Letterman will sign off fromThe Late Showfor the final time after 33 years in late night, and no one is sadder about him leaving than Letterman himself.

"I'm awash in melancholia," Letterman tells the New York Times. "I'll miss it, desperately. One of two things: There will be reasonable, adult acceptance of transition. Or I will turn to a life of crime."

Still, he promises that his last show, will be "cheery." "I want it to be upbeat, and I want it to be funny, and I want people to be happy that they spent the time to watch it," Letterman tells the Times. "Johnny [Carson]'s last show was historic. This one won't be. [Laughs] This one, people will say, 'Ah, there you go. When's the new guy starting?'"

See the star-studded guest list for Letterman's final shows

The new guy, of course, is Stephen Colbert, who's scheduled to take the reins later this year. Letterman said he was not part of the decision to hire Colbert as his successor, but certainly held opinions about who should take over and was initially miffed that CBS didn't consult him. "I always thought Jon Stewart would have been a good choice," he tells the Times. "And then Stephen. And then I thought, well, maybe this will be a good opportunity to put a black person on, and it would be a good opportunity to put a woman on. Because there are certainly a lot of very funny women that have television shows everywhere. So that would have made sense to me as well."

Though he says he hasn't offered any advice to Colbert, Letterman is confident that the Colbert Report host "will add a third, different dynamic" against Jimmy Kimmel and Jimmy Fallon. But how did he know it was time to go? Letterman tells the Times that this actually wasn't the first time he considered leaving The Late Show (and not always by choice). Here are seven times he almost walked out the door:

David Letterman announces he's retiring from The Late Show

1. When he went head-to-head with Jay Leno.
Asked if he thinks the rivalry between himself and Leno was "overblown," Letterman says it wasn't. But he says he approached the contest with little confidence. "Before, I felt pretty confident in what we were up to, because there was no competition to speak of," he says. "We prevailed for a while, and then I lost my way a little bit. Quite a little bit. And at that point, there was not much I could do about it. People just liked watching his show more than they liked watching my show." He expected to be given a pink slip as a result. "It couldn't be more clear that Jay was the more popular show. And when we all realized that there's not much we can do here -- you can't put toothpaste back in the tube -- then we started going our own way again," he says. "I was always surprised that they didn't let me go. ... Next thing you know, I've been here 23 years."

2. When Howard Stringer left CBS.
After Stringer brought The Late Show over from NBC, he spearheaded the remodeling of the Ed Sullivan Theater. But when Stringer left, the future of CBS was unclear. "Those were the days when CBS was really doing poorly," Letterman reflects. "They lost [N.F.L.] football. And I just thought to myself, this can't be good for us. Then [Les] Moonves came in and turned the place into Disney World."

Jay Leno weighs in on David Letterman's retirement

3. When he had heart surgery in 2000.
Letterman says his biggest fear after his quintuple bypass was that he wouldn't be able to run again. But there were also professional concerns, as CBS enlisted various other celebrities to take over hosting duties, including Regis Philbin and Janeane Garofalo. "I was worried that somebody would go on ... and be good enough that they didn't want me back," Letterman tells the Times. (Also, he adds: "Six weeks after the surgery, I ran for five miles. So let's face it, I am a hero. There's no two ways of looking at it.")

4. When he was blackmailed in 2009 about his sexual relationships with female staffers.
Speaking candidly about his "sex scandal" (his words), Letterman notes: "Looking at it now, yes, I think they would have had good reason to fire me. ... But they didn't. So, I owe them that." For his part, Letterman claims he was "ignorant" about the ramifications of his actions. "It just seemed like, O.K., well, here's somebody who had an intimate relationship with somebody he shouldn't have had an intimate relationship with," he says. "And I always said, 'Well, who hasn't?'"

5. When he realized the Internet wasn't his friend.
Letterman, 68, says he feels that the new era of late-night shows, marked by videos that go viral the next morning on the Internet, "just came and went without me. It sneaked up on me and went right by." Despite his staff members encouragements, Letterman has steadfastly refused to join Twitter. "I recognized the value of it. It's just, I didn't know what to say," he admits. "You go back to your parents' house, and they still have the rotary phone. It's a little like that."

Finale Preview: Get spoilers on 35 must-watch season-enders

6. When he watched John Mayer host an episode ofThe Late Late Show.
The realization that the late-night landscape has changed in recent years further hit Letterman when he watched Mayer helm The Late Late Showearlier this year, before James Corden took over for Craig Ferguson. "He's young. He's handsome. He's trim," Letterman notes. "So then I realized, I got nothing to worry about. I know I can't do what Jimmy Fallon's doing. I know I can't do what Jimmy Kimmel is doing. There's nothing left to be worried about. It's all over, Dad, you're going to be just fine. You're going to a new place. They'll be very nice to you, Dad. You'll make a lot of friends."

7. When Jay Leno leftThe Tonight Show.
Letterman says he found a sort of solidarity with his NBC rival. When Jay was on, I felt like Jay and I [were] contemporaries," Letterman says. "I thought, this is still viable -- an older guy in a suit. And then he left, and I suddenly was surrounded by the Jimmys."

Read Letterman's full interview with the New York Times here.