The former England manager wowed the judges with his Quick Step as the Devil in Halloween Week, but it was his score of four from the week before that was on his mind.
Adams, 56, said: "I've got to say to Shirley, I ain't got a clue, but I've got to apologise for my comment. I was out of order. Dont go home Shirley, because you are Strictly."
In Week Five the retired footballer and his partner Katya Jones performed a Cha-Cha-Cha to the Grandstand theme tune and he made multiple missteps.
But when Ballas held up her '4' paddle Adams was seen on camera bellowing: "Shirley! Go home!"
Asked about his outburst on spin-off show It Takes Two defended himself, saying: "When they started making comments I got a bit reactive. It is no disrespect to them it is just my way of going 'Go away.'
"I'm a beginner and at times I don't do the right moves and stuff."
Adams has consistently been near the bottom of the score board for his performances so far, but escaped the dance-off after proving popular with viewers.
But this week the judges said his Quick Step proved he was not the token 'comedy contestant' of the series, having proven he can actually dance.
Ballas - dressed up as Gomez Adams - told him: "Who would have thought at Strictly o'clock on Halloween you would bring that sort of performance? I am in shock. Your footwork was lovely. You have absolutely shocked me."
Craig Revel Horwood, in drag as Morticia Adams, said: "Welcome Tony to the ballroom darling! I think you have found your dance. Brilliant!"
Anton Du Beke called him "Unrecognisable," and said he was no longer pigeonholed as the comedy contestant.
Du Beke said: "You are a proper dancer."
Motsi Mabuse recalled scoring him four in previous weeks and said: "I'm like look at this man now! What a surprise, what hard work!"
Adams scored 31/40, his highest of the series.
Ballas, 62, previously appealed for "respect" after being trolled by viewers over her judging critique of dancers.
Watch: Tony Adams reveals tense team talk with Strictly partner Katya Jones.