Dylan and Cole Sprouse Once Brushed Off Matt Damon When He Visited The Suite Life Set: ‘Ridiculous’

Matt Damon’s star power wasn’t enough to get Cole and Dylan Sprouse to put down their controllers as teens, however famous he may be.

On a recent episode of SiriusXM’s Let’s Talk Off Camera with Kelly Ripa, Cole recalled how the twin brothers’ childhood obsession with the World of Warcraft video game led to them snubbing the Oppenheimer actor, 53, when he visited them on set.

The Riverdale alum, 31, revealed that he and Dylan were playing the video game in set school when one of the production assistants announced that Matt and his kids were big fans of their show, The Suite Life of Zack & Cody, and would be popping in to say hi.

“I remember Dylan and I turning to each other and just going, ‘Ugh. I can’t believe we have to get off a World of Warcraft right now,'” he laughs, adding that they were around 15-years-old at the time.

Sure enough, when Matt announced his arrival with a knock, Cole said that every crew member stood in the hall “just looking and amazed at Matt Damon” while he and his twin never turned away from their screens to greet him.

“This was how ridiculous we were as children,” he continued. “I think he came in and was like, ‘Hey guys. Wow, so nice to meet you,’ and we were like, ‘Yeah, yeah. Hey man. Hey, how you doing?’ and just went right back to the game.”

“We were just being kids and playing video games, and I’m sure if I ever meet Matt Damon again, I’m just gonna say ‘I’m sorry’ and it’ll be all good,” the Lisa Frankenstein actor added.

Elsewhere in their conversation, Cole opened up about another instance where he and Dylan wreaked havoc while attending school on set.

Admitting that they were “notoriously bad” troublemakers for the Student Teacher Union, Cole recalled how the adults running the set school were “terrified of us” after they performed a series of pranks.

After meeting a teacher named Laura Torrance that the two “fell in love with” on Big Daddy, Cole and Dylan wanted her to be on set with them for their Disney Channel show.

“They kept saying, ‘No, no. We can’t send Laura. We can’t send Laura, blah, blah, blah,'” he recalled before revealing that they created a “hit list of teachers” and drew caricatures of the ones who would come in.

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

“We would, underneath their portraits, mark how many days they stayed with us,” Cole said. “We were horrible. I mean truly devils.”

When the head of the Student Teacher Union was sent to control the twins, she lasted “one day, and the very next day we got Laura.”