Surf Life Saving New Zealand regional manager for the eastern region, Chris Emmett, said the tragedy unfolded in a matter of seconds.
"The gentlemen was breaststroking at the time and then he put his hand up."
One lifeguard swam out to the man while another radioed for an inflatable rescue boat. "They were with him within 45 seconds of getting there."
Mr Emmett said most people who called for assistance could hold themselves up for several minutes but the 110kg man was unconscious when pulled from the water. The lifeguards performed CPR until paramedics arrived, but the man was pronounced dead at the scene.
Mr Emmett said the lifeguards did everything right despite the outcome.
The man's death was not inside the official Christmas holiday period which began at 4pm on Christmas Eve.
Mr Emmett urged beach-goers to swim between the flags and research beach conditions online before visiting.
"We need to keep going back to one of our core messages - always swim between the flags."
New Zealand had 86 patrolled beaches with 22 in the eastern region, where professional lifeguards monitored the beach from 10am to 6pm.
He said Whiritoa beach was exposed, dropped off steeply and that day the water was affected by the end of tropical Cyclone Evan with 2.5m swells.
Whangamata Police Constable Russell Richards said the man's brother had decided not to go for a swim during their nostalgic visit at lunchtime on Monday.
He said police were still notifying the family of the man, who was single and did not have any children, of his death.
In February last year Rotorua school teacher Sarah Nicholson was caught in a rip at Whiritoa and drowned.
Key messages for people at the beach this summer were to be prepared, watch out for yourself and others, be aware of the dangers and know your limits.
For more information about beach conditions visit www.findabeach.co.nz
