"A local boat had pulled him on board and brought him to shore, but he had already passed away."
It is not clear how the man died.
Police say it is understood the man was diving with a companion at the time.
Martinborough Fire Brigade chief Bill Butzbach said one appliance attended to assist police and ambulance staff.
"Sadly, it was a fatality and there wasn't much we could do."
He said he believed the man lived and worked in Masterton.
A resident from the area, who wished to remain anonymous, said plenty of people went diving off the coast, but there could be strong tides in the area.
"There was a bit of wind [yesterday], but conditions were fairly normal.
"I saw the helicopter go past a couple of times . . . it's usually visitors that get themselves in trouble, but today was a fairly calm day."
Police were called to the scene just before midday, but a statement said they could provide no further information due to limited communications in the area.
Wellington Free Ambulance spokesperson Diane Livingston confirmed paramedics had attended the scene, but also said she had little information about the circumstances.
In 2013, Lynne Falk, 64, died while snorkelling for paua with her partner near the same area.
Carterton man Michael Siemonek, 49, died while paua diving at Tora in 2015.