Speaking to Reuters on Tuesday, Brendan Schwab said his organization had a number of areas of concern over protocols designed to protect athletes and other people attending the games, which are due to take place in July and August.
The IOC and the Japanese government are facing increasing pressure to cancel or postpone the Olympics with Covid-19 cases rising in Japan and other parts of the world.
The IOC says measures it has put in place, including mass testing for coronavirus and keeping interactions to a minimum, will ensure the games can go ahead smoothly.
However, Schwab questioned whether the testing protocols were strict enough.
"We don't think the IOC is listening, which is really unfortunate," said Schwab. "It's too early to say that the games should be cancelled. There's still two months or so to go. But we do believe that there needs to be an urgent commitment to setting in place the very best COVID-19 protocols that we saw in professional sports. The clock is in certainly ticking on that."