The Tokyo 2020 Olympics committee President, Yoshiro Mori, has now resigned over the sexist remarks he made that sparked outcry at home and abroad.
Mori apologized, again, and stepped down at a meeting on Friday.
“The important thing is for the Olympics to be held in July and in order to hold the Olympics, if my presence is a hindrance to the preparations, this cannot be allowed."
The former Japanese prime minister started the controversy when he said women talk too much, while he led a meeting earlier this month.
He initially refused calls for his resignation and apologized at the time, but the pressure became too much after a second sexist remark.
It's not immediately clear who will replace him.
Mori had asked the mayor of the Olympic village, Saburo Kawabuchi, to take his position, but local media report that the mayor has declined.
There had also been some public criticism at the prospect of Kawabuchi taking over, being another older man -- 84 years old.
One Reuters source has said that many officials want a woman to replace Mori.
Local media report that Olympics Minister Seiko Hashimoto is being considered.
Mori’s resignation comes just months before the postponed Summer Games are scheduled to begin.