Seinfeld’s Michael Richards Says Racist Laugh Factory Rant Made Him Face His Insecurities: ‘The Damage Was Inside of Me’ (Exclusive)

On April 30, Seinfeld fans were surprised to see Michael Richards, the actor who played Cosmo Kramer on the sitcom between 1989-1998, at the premiere of Jerry Seinfeld’s new movie, Unfrosted.

Because for the past 18 years, Richards has largely stayed out of the public eye — ever since he was caught on camera in 2006 hurling racial insults, including the N-word to a group of hecklers during a stand-up set.

Now he’s making a tentative return to the public eye with the release of his memoir Entrances and Exits (out June 4 from Permuted Press), in which he writes about his childhood, his rise to fame on Seinfeld, and yes, that night at the Laugh Factory that upended his career.

“I was immediately sorry the moment I said it onstage,” Richards, 74, tells PEOPLE. But he knows he doesn’t expect the world to forgive and forget. “I’m not looking for a comeback.”

“My anger was all over the place and it came through hard and fast,” he continues. “Anger is quite a force. But it happened. Rather than run from it, I dove into the deep end and tried to learn from it. It hasn’t been easy.” He adds, “Crisis managers wanted me to do damage control. But as far as I was concerned, the damage was inside of me.”

Richards says he’s spent the past 17 years in “deep analysis.” “It was time to figure out where all the anger was coming from,” he says. Part of it came down to his own insecurities, and a feeling of not being wanted. (He was raised by a single mom who had initially wanted an abortion, but they were illegal and dangerous at the time.)

“Somehow I couldn’t connect to the joy of being an artist,” he says, of becoming famous as Cosmo Kramer on Seinfeld. “I was a good character actor, but I was comfortable being the character, not in being me.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

That insecurity led him to turn down opportunities. “I said no to the offer of a star on the Hollywood Walk of Fame. I didn’t feel deserving,” he says. “I said no to hosting Saturday Night Live twice because I didn’t feel good enough. I was never really satisfied with my Seinfeld performance. Fame magnified my insecurities.” 

Of the night in 2006 when he fought with a group of hecklers, he has no valid excuse. “I’m not racist,” he says of the hurtful words he used that night. “I have nothing against Black people. The man who told me I wasn’t funny had just said what I’d been saying to myself for a while. I felt put down. I wanted to put him down.”

However, Richards’ memoir isn’t only about that night his career ended; it’s also about his unconventional upbringing by a single mom, the trauma over learning the truth about his father, his time in the Army, his rise to fame after becoming Cosmo Kramer and his difficulty with being a celebrity. It’s also about the interpersonal work he’s done and his life over the past 17 years, mostly spent reading and studying religion and philosophy.

“[I’m] learning and healing. Healing and learning,” he says of his life now, as husband to actress Beth Skipp and dad to their son Antonio. “But life is always an up and a down. I continue to work through the day and the night, the light and the dark that I am.”  

To read more about Michael Richards, pick up the latest copy of PEOPLE on stands Friday.

Entrances and Exits will hit shelves June 4, and is available now for preorder, wherever books are sold.