Matthew McConaughey Opens Up About His ‘Small Town’ Childhood with 'Rebellious' Parents and Bankruptcy Scares

“My dad was trying to keep from going bankrupt at times,” the Oscar winner recalled on the latest episode of the ‘Good Trouble with Nick Kyrgios’ podcast

Matthew McConaughey is looking back on some of the more unique facets of his "small town" upbringing.

“I was born in a little small town, Uvalde.” the award-winning actor, 55, told tennis pro Nick Kyrgios on the latest episode of the Good Trouble with Nick Kyrgios podcast on Wednesday, Nov. 20. "We left in 1979 because it was kinda getting too big. It was 12,000 people and a Wendy’s burger moved into town.

McConaughey, who moved back to his home state in 2014 after years spent in Hollywood, explained there was an “oil boom” in Texas at the time. His father moved the family to the town of Longview in hopes of getting a piece of the profit — but it created periods of financial uncertainty for the family.

Related: Matthew McConaughey and Wife Camila Reveal the Exact Moment They Decided to Leave L.A. and Move to Texas

“We were middle class. We got by, I think, it was tough. My dad was trying to keep from going bankrupt at times," McConaughey recalled. "But he went from truck driver to running a gas station to being a pipe salesman, and we started to roll a little bit. As he said, he started to ‘hit licks’ in the '80s and we started to roll.”

Related: Matthew McConaughey Calls for Change After School Shooting in Texas Hometown: 'We Must Do Better'

As for his mom, he says, she was a teacher who liked to bend the rules.

“My mom was the school teacher who would actually come home and go, ‘check it out, here’s the answers to the test.’ She’d be like, ‘this is a stupid curriculum, don't even read it,’” he explains with a smile.

“My mom and dad were rebellious in that way. They were real good trouble, man. They were outlaws,” he says, adding, “We were a resilient family.” 

Related: Matthew McConaughey Reveals the ‘Pact’ He Made with Wife Camila After Leaving Hollywood for Their Texas Ranch

Despite making it big, McConnaughey found his way back to Texas decades later when he and his wife Camila Alves McConaughey, 41, agreed it would be best for their family to put down roots there again.

The married couple of 12 years share three children together: sons Levi, 16, Livingston, 11, and daughter Vida, 14.

In March, the couple opened up about their decision to leave their “happy life” in Malibu, Calif. to live a slower lifestyle in Texas during an interview with Southern Living.

The pair were visiting Matthew’s mother when Camila said she couldn’t help but notice the “gravity” that the state held for her husband throughout their trip. While stopped at a red light, Matthew said he remembered Camila "blurted" out, "You want to move here, don’t you?” And the rest was history.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories. 

After officially making the move with their kids in 2014, the two said they’ve noticed the positive impact living outside of Hollywood has had on their family.

“Time slowed down,” Matthew told the outlet. “The clock was right, the body clock. And part of that is ritual; part of that is just the distance between places and the way people move. But it’s also the hospitality, the courtesy, the common sense, the lack of drama.”