Three Coastguard vessels and the police helicopter searched for the man for nearly four hours in choppy conditions.
His overturned dinghy was found close by.
The man was wearing a wetsuit and lifejacket, which his rescuers credit for keeping him alive.
He was rushed to a waiting ambulance at Mangere Bridge.
"He's a very lucky man," said Rod Frost, who was the skipper of the volunteer Papakura coastguard crew who found the man.
"There were some rough conditions out in the Manukau Harbour, and although we were searching much of the night, we only spotted the man when our searchlights picked up the reflective strips on his lifejacket.
"The lifejacket kept him afloat throughout the night, and that undoubtedly saved his life."
However the man was unprepared, having only oars and a lifejacket and no forms of communication to signal if he got in trouble.
Coastguard urged boaties heading out on the water to wear a lifejacket, carry at least two forms of waterproof communication, and file a trip report with Coastguard to aid search and rescue teams.