The three men were all wearing lifejackets and would not be alive now if they did not have them on, they said.
Their next lucky break came three hours later when No Xcuse came into view. Despite a number of fishing boats and other vessels out, nothing had come close enough to see them.
"We were getting cold, were cramping and tired. Every time one of us tried to swim towards shore we got pushed back by the swell."
No Xcuse skipper Charlie Destounis said the two older men were totally exhausted.
"They had nothing left. One of them was swallowing water. They would not have lasted much longer."
The No Xcuse crew were on their way back to the wharf after a day fishing.
"We were cruising in, filleting the fish, and I looked out the window and saw a chilly bin floating. Then I saw the bow of a white boat sticking out of the water," said Mr Destounis.
A quick scan and he saw three hands coming out of the sea.
"The were not far from the boat. One was hanging on to a rope, the other two were drifting away. They were not in a good way."
They were too exhausted to get themselves on to the boat and several of the No Xcuse crew had to haul them in.
Mr Destounis said there were a number of "uncanny" events that led them to take a route near the buoy at that time.
"We were going to fish for another couple of hours - it was lucky we pulled the pin when we did or those guys might not have been above water."
Because of the weather and swell, the skipper took a different angle into port.
"It was lucky we came in at that angle - it was all a bit uncanny really."
Also on the rescue boat was Bill Destounis.
"They're lucky boys. Lucky to still be alive," he said.
The men had a radio on board their boat, but "they told us it wasn't working", Mr Destounis said.
Lion Foundation Coastguard skipper Mike Schultz said the boat went down so fast a radio might not have been much use to them.
The Lion Foundation Coastguard boat was called out to recover the sunken craft.
"When we arrived, just the bow of the fibreglass boat was visible above the surface of the water," said Mr Schultz.
The Coastguard crew were able to attach a tow line to the bow.
"We were able to tow the boat up out of the water, then we towed it ashore for the owner to take charge of it."
- Gisborne Herald