Simon Cowell, The X Factor Simon Cowell, The X Factor

The X Factor burst onto TV hoping to be the brighter, bolder brother of American Idol. But ratings haven't been as big — The X Factor has averaged 12.7 million viewers this fall compared to American Idol's 25 million last season. Still, exec producer Simon Cowell says viewer awareness is up, helped by the show's surprise eliminations and unscripted drama.

"In the U.K., it took us seven seasons to really get to a point where I was happy," says Cowell, who created the show in England. After one season in the U.S., "I'm happy with the talent, I'm happy now with the buzz, and I think people understand the show."

He admits there were a few hiccups this season, specifically the failure of Paula Abdul's groups. "Paula made a big mistake," says Cowell. "She should have kept that kids' group [inTENsity] in. Going into the second year, hopefully we'll get better groups."

There has also been intense controversy and anger surrounding decisions made by judge/mentor Nicole Scherzinger. After voting off Drew Ryniewicz and then forcing a deadlock that sent Rachel Crow home, viewers have been viciously attacking Scherzinger through Tweets ("You are a lowlife for voting off Rachel!"), Facebook messages and even death threats. "It's ridiculous, but I've had them in the past and I'm still alive," says Cowell. "I respect the fact that we have rules, and you can't cry about it afterward because that's the way it is."

Will he replace Scherzinger next season? "Haven't got a clue," he says. "We change these shows all the time. You listen to the network, you listen to the sponsors."

Ultimately, Cowell believes they've fulfilled their mission statement. "We came over here to find a star, and I think we found a few," he says. "People now get the rules and understand that we encourage older singers and teenagers to compete. I do believe we're part of a next-generation talent show."

Subscribe to TV Guide Magazine now!