"He beat my ass pretty good a bunch of times. ... I just remember, like, clinched teeth, strained-neck look on his face," James said during his Nightline interview Tuesday. "I was a terrorized kid, and, I mean, it's really tough for me to think about now because ... [my daughter] Sunny is the age that I was when my dad broke my arm."
James' father, Larry, has since sent ABC News an email calling the allegations of abuse false.
The Monster Garage star also explained why he believes the alleged abuse played a role in his extramarital affairs.
"I don't think I have a sex addiction where I'm running around trying to have sex with everyone and I can't stop," the 41-year-old said. "There's a lot of people who are abused that go on to abuse, or go on to do drugs... or go on to, you know, cut themselves. ...They'll do it to self-sabotage their life ... because of their abuse, they've never felt good enough.
"Why would I totally destroy my wife's life and humiliate her when I knew it was bad, you know?" James continued. "I may seem like a monster in peoples' eyes, but I'm not that kind of person where I will willfully, you know, 'I'm gonna get her, I'll show her.' That's not what it's about. It's about trying to push someone away that I thought was going to leave anyway."