Why James Remar Doesn’t Believe His Sex and the City Character Was the Worst Boyfriend on the Show (Exclusive)

James Remar doesn’t think his Sex and the City character, Richard, was the worst boyfriend on the show — despite what some fans may say.

Remar spoke to PEOPLE exclusively while attending the Dexter: Resurrection world premiere at Alice Tully Hall in New York City on July 9. During the conversation, the actor revealed his thoughts on the on-again, off-again relationship between Richard and Samantha, played by Kim Cattrall.

“I’m the best boyfriend on the show,” says Remar, 71, who then recalled a talk between him and Michael Patrick King, a series writer and director.

“I said, ‘How come you dissolved our relationship? It’s like we were the two that were the best for each other.’ And he said, ‘We only introduce love on this show to have it fail,’ “ Remar adds.

“So at least he said that there was love there,” he continues. “And he said, ‘But it failed and you’re not coming back.’ “

When asked if he would ever consider returning to the series, Remar said he’d honestly be pretty surprised if he’d be asked back.

“I am sure it’ll be difficult to say no, but I’m an old guy. They’re not going to call me,” he says.

The Oppenheimer actor recently reprised his role as Harry Morgan — Dexter’s late adoptive father — on the Dexter reboot series, Dexter: Resurrection. While speaking about the role, Remer said it’s “totally new and completely different at the same time.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

“Working with Michael [C. Hall] has always been a privilege and a very, very intense experience because he’s so focused and so precise that you have to bring your best, you have to bring your A game. You can’t fool around and flub your lines and do stupid stuff,” he says.

“When you’re working with Michael C. Hall, you got to know your lines and you got to do the right thing because he always does, and he’s doing it every single scene. And I come in and do a scene. I go home and Mike’s there all the time. So that part hasn’t changed,” he says.

The PEOPLE Puzzler crossword is here! How quickly can you solve it? Play now!

“The surrounding venue, I mean, we’re in New York City, so that’s very different,” he continues, referring to the fact that the original series took place in Miami and was filmed in L.A.

He also added that both he and Hall have allowed their characters’ relationship “to evolve” in the new installment.