The Birmingham and Yardley MP, who is in the race to succeed Jeremy Corbyn, said the Labour Party had a "blindspot" when it came to women leaders.
Asked whether men should step aside to allow women to come through, she added that "passing the mic" can be "the greatest way to show you truly believe in something."
She said: "It's genuinely frustrating. It does a disservice to the Labour Party's record on women.
"I think it would look bad, to be honest."
Asked if it would be an "embarrassment" if Sir Keir - the only male candidate and current favourite - won, she said: "Yeah, I think it would be embarrassing.
"When there's one man in the race and four women... "It will be embarrassing."
Sir Keir and Ms Phillips are in the running alongside Rebecca Long-Bailey, Emily Thornberry and Lisa Nandy.
She added: "It's not great if we can't ever seem to think that the women are good enough.
"When you're a woman in politics, you're always "the next time". It's always "we have to pick the best person for the job" - they often mean the best man and that is disappointing."
Asked if men needed to "step aside" to allow women to come through, she replied: "If you truly believe in women's representation.... sometimes passing the mic is the greatest way to show that you truly believe in something.
"But it's a very difficult argument to say you should stand back if you truly think you're the best person for the job."