The actor, 31, rose to fame as Anwar Kharral in the Channel 4 series, which he appeared in from 2007 to 2008.
He told the New York Times that when he was appearing in the TV show he used to go into chat rooms and read what people were saying about him and that it was “brutal”.
“There were all these lists of who’s the favourite character on the show or who was the best-looking character, and I was always the ugliest, the least attractive,” he said.
“No one liked Anwar. It really took a toll on me personally.”
Patel has since gone on to star in big films such as Slumdog Millionaire, Lion and The Last Airbender.
He was nominated for an Oscar for 2017’s Lion, in which he starred as a boy who is hunting for his home after getting separated from his family in India and taken to Australia and adopted.
However, the actor told the publication that he had low self-esteem and didn’t feel “worthy” of the nod.
Patel was up against Mahershala Ali, Michael Shannon, Jeff Bridges and Lucas Hedges for the award.
“You're there with really impressive creatures, the best of the best, and you're like, 'I don't know what I have to offer in this space’,” he said.
Patel has a number of projects in the pipeline including medieval fantasy film The Green Knight and action thriller Monkey Man.
Watch: The Green Knight - Teaser trailer