Hugh Hefner and Holly Madison Hugh Hefner and Holly Madison

Even Hugh Hefner can suffer from a broken heart.

The Playboy mogul has confirmed rumors that he and Girl Next Door Holly Madison are splitting up. "If she says it's over, it's over," Hefner told E!. "But like I've said before, she is the love of my life, and I expected to spend the rest of my life with her."

Hefner, 82, said that the beginning of the end came six months ago, when he learned that he could no longer father a child, thanks to low sperm count. "She'd like to be married and have children, but it's not in the cards here for me," he said. (Insert collective "Awww!").

While there are two more ladies left in Hef's life — Kendra Wilkinson and Bridget Marquardt — the romantic relations continue, but it seems his heart was set on Madison.

Of course, don't cry for Hef for too long. Being the world's most famous ladies' man has its perks: Last week, he was seen with 19-year-old twins in tow, plus there's also a college senior from Georgia already testing the girlfriend waters.

The plethora of appealing options is not lost on Hefner. "I'm dancing as fast as I can," he laughed. The loss of true love, though, is a heavy burden — even at the Playboy mansion. Is Hef's breakup just a little tragic, or just a little bump in the romantic road?