Footage carried by local media showed rescuers carrying out two bodies and one injured passenger from the crash site on Friday afternoon.
Taiwan's premier, Su Tseng-chang, visited the crash site.
The crash, north of the eastern city of Hualien, killed the driver of the train carrying many tourists and people heading home at the start of a long weekend traditional holiday to tend to family graves.
Taiwan media said many people were standing as the train was so crowded, and were tossed about by the crash impact. Media showed pictures of survivors being led out of the tunnel.