At least one passenger said the train was going faster than normal as it moved along a curve on a downward slope leading into Spuyten Duyvil station, just north of Manhattan.
Before reaching the station, the derailed wagons flew across a grassy bank separating the rail line from the Hudson and Harlem rivers, which meet at that point.
All seven of the train's cars came off the tracks and the front carriage came to rest only a few feet from the river. Two carriages toppled on their side and two others also completely left the track, barely staying on their wheels.
More than 130 firefighters were quickly on the scene using air cushions to stabilize the wagons.
Divers searched the rivers in case passengers were hurled into the frigid water by the force of the derailment. Three of the dead were thrown from the train, police said.
Police sniffer dogs remained at the wreckage hours afterwards, making sure no casualties had been missed.
Philip Banks, a top New York police official, told reporters more than 100 people were believed to be on the train which had left the town of Poughkeepsie at 6:00am (1100 GMT). The accident happened 80 minutes later.
A freight train derailed on a nearby stretch of track earlier this year.
But passenger Frank Tatulli, who escaped with injuries to his head and neck, told WABC TV that the commuter train was going "a lot faster" than normal as it went into the bend.
Other travelers gave similar accounts about the speed. The unnamed train driver survived the crash, police said, and was being treated in hospital. He was to be questioned by accident investigators later Sunday.
The curved section of track leading into Spuyten Duyvil is a slow speed area. Thomas Prendergast, chairman of the Metropolitan Transportation Authority which runs the regional rail and bus network, said speed would be probed.
But New York state governor Andrew Cuomo told reporters at the scene: "We need more information before we can make any hard determinations."
The rail network "has had a number of accidents in the last nine months, and so we are very concerned about seeing another one," National Transportation Safety Board chairwoman Deborah Hersman told CBS television.
The NTSB said in a statement that its investigation team would include specialists in track, signals, mechanical systems, operations and human performance. A press briefing about the probe is expected later Sunday, though no time was announced.
Sunday marks the end of America's Thanksgiving holiday weekend and a surge of travelers was expected.
The White House said that President Barack Obama had been briefed on the accident. Governor Cuomo said the deaths had cast a shadow over Thanksgiving.
"Let's all say a prayer and remember those people who we lost this morning and their families and we'll let the first responders do their job."
- AFP
