Landslides triggered by heavy rains hit the central Japanese city of Atami on Saturday (3 July).
At least two people have been killed and many more are still missing, according to public broadcaster NHK.
Prime Minister Yoshihide Suga convened an emergency task force to tackle the crisis.
He assured the public that rescue efforts were ongoing, and encouraged those in the area to stay alert.
The floods are a reminder of the natural disasters that plague Japan, including earthquakes, volcanoes, and tsunamis.
The country's capital Tokyo is to host the summer Olympics starting this month.
Drone footage aired by NHK showed collapsed and half-submerged houses, as water dashed the city with mud and debris.
Social media images showed partially sunken cars and rescue workers wading through waist-high water with a small life raft.
Japan's military sent emergency rescuers to the city, where NHK said about 80 people were evacuated.
Tokyo Electric Power said thousands in the area are also facing power outages.