The TV star has been on the panel of the ITV entertainment show since 2012 but The Sun quoted a source as saying: "He hasn’t taken the decision lightly, but it just feels like time to move on."
It comes just weeks after Walliams, 51, apologised for making derogatory remarks about contestants on the show.
Read more: David Walliams 'upset' by phone hacking
A spokesperson for the programme told the PA news agency: “The judging panel for Britain’s Got Talent 2023 will be announced in due course.
“It’s still very much up in the air at the moment on whether David is going to take part in next year’s show. No decision, though, has been made as yet.”
All three are expected to be back on the panel when the series returns, but there has not yet been an official announcement.
Speaking on Good Morning Britain, presenter Richard Arnold pointed out that with auditions still two months away, no decisions would had been made and the stars would not yet have signed contracts, including Simon Cowell.
Holden recently said she'd like to see Walliams return.
Talking to Good Morning Britain at the Variety Club Showbusiness Awards, she was asked how she’d like to see the judging panel look.
“I would like to see the judging panel look like it did before,” she said. “I think we’re a great team, we’re a family.”
Asked if she was "Team David", she replied: "Of course I'm Team David."
Walliams, 51, found himself in hot water earlier this month when the Guardian reported that he had been overheard making insulting comments about some of those auditioning while on the BGT set.
The children's book author was heard referring to a pensioner as "a c***" repeatedly, and saying of a female contestant: “She’s like the slightly boring girl you meet in the pub that thinks you want to f*** them, but you don’t."
Walliams apologised, saying in a statement: “I would like to apologise to the people I made disrespectful comments about during breaks in filming for Britain’s Got Talent in 2020.
"These were private conversations and — like most conversations with friends — were never intended to be shared. Nevertheless, I am sorry.”
Yahoo has contacted ITV and representatives for Walliams for comment.
Watch: Susan Boyle lands top spot in poll of best Britain's Got Talent moments