The 56-year-old soap star - best known for playing Heather Trott in Albert Square - married Yassine Al-Jermoni, 35, near Agadir, Morocco in 2011.
Fergison told the Mirror: "My son Alex asked his Alexa: ‘Tell me a fact about Cheryl Fergison’ And Alexa goes: ‘Cheryl Fergison is married to a goatherd from Morocco.’ Why’s that the first fact that comes up? And it’s wrong! We don’t have any goats. We’ve never had any goats near us! But that’s the first thing people say. I find it really difficult to pin down why.
"There’s racism, sexism, ageism - all the isms are there when people talk about our relationship.
"The world has become so cynical. People think it’s got to be a con, it’s got to be a problem. But I say, ‘It hasn’t got to be anything. Just stop it!’”
The actor claimed women who marry younger men are treated differently to men who marry younger women.
She said: "It seems to be quite cool for the men, like Mick Jagger, to date younger. And it seems okay for them to have someone else two years later. But people don’t give any kudos to a woman dating someone younger, even though we’ve been together 11 years.
"Well we don’t care. It really is absolutely water off a duck’s back to us now. I just feel sorry for them, that they need to criticise our life."
Fergison starred in EastEnders as obsessive George Michael fan Heather between 2007 and 2012, before her character was murdered by Ben Mitchell. She returned to record a voice over as Heather in 2016.
She has previously said she would like to return to the soap as Heather's long-lost twin.
She posted on social media in 2020: "Ok so I’ve had lots of people asking if I could go back to @bbceastenders as Hev's long lost twin... so writers there's a challenge and actually there is a storyline that could defo make that possible...
"Remember Hev's mum ( she was a wicked one) and aunt babes terrible past story of selling babies !! ...if you would like to see this epic come back let @bbceastenders know ..."
Watch: Tribute to late EastEnders star June Brown appears on London wall