The former tennis star, who has just retired from her role hosting the BBC's coverage of The Championships at Wimbledon, wrote about her frustrations at his comments that he "didn't love her quite enough to marry her" in her new book which was revealed by Mail Online.
Sir Cliff spotted Barker while she was celebrating after a tennis tournament in Brighton with his Shadows bandmates in 1981 and asked her on a date.
Although at the time Barker was only 25 while Sir Cliff was 41, they enjoyed a brief relationship of less than a year with Barker saying they "hadn't taken it far". but after a period apart due to their hectic schedules Sir Cliff stopped contacting her and it fizzled out.
They have stayed friends, but Barker wrote: "The only thing we fell out over was the fact that he kept harping on about me in interviews — ‘I didn’t love her enough to propose,’ and so forth."
Barker, now 66, said that the comments had "really upset" her and that she had been desperate for Sir Cliff to meet someone else so that he would stop mentioning her.
Read more: Bake Off unveils the 2022 contestants
The former Question of Sport host wrote: "If someone had told me that my relationship with Cliff would last a few months but I’d still be hearing about it 40 years later, I wouldn’t have gone near him. Frankly, it looks silly now that he’s still talking about a relationship that was never really more than a friendship."
She married Lance Tankard, a Met Police detective, and said that the interviews had been upsetting for him, too.
The couple had woken up to the newspaper headline "Why I Couldn’t Marry Sue — by Cliff" the morning after their wedding and Barker says she has "begged" Cliff to respect her marriage by stopping the comments, even appealing to his agent and manager over the years.
Adding that "it has been frustrating", she wrote that the idea of her "waiting at the altar for him" was completely false.
She added: "Yes, I really enjoyed our early friendship, but continuing to harp on about me simply isn’t fair, not just to me but to Lance, who has been constantly reminded why someone else wouldn’t marry his wife. It’s not respectful."
Watch: Emotional Sue Barker receives standing ovation during Centre Court celebration