"Sadly, he died on the scene," Mr Brettkelly said.
Police said the cause of Mr Smith's death was still being investigated.
"Our thoughts are with his family and friends at this difficult time," a spokeswoman said.
Michael Pugh swam the 2.6km race that morning with his daughter and was returning to the beach to watch his son swim in the next race when he saw the tragedy unfold.
An announcement came over the loud speaker at the event that someone had gone missing.
"The IRBs raced out to do CPR on him. They were doing CPR as they landed on the beach.
"I saw him get carried up."
Mr Pugh heard Mr Smith's partner scream. He said the police escorted Mr Smith's partner to where he was lying and set up a tent to keep everything private.
Mr Pugh believed Mr Smith's partner had raised the alarm when he did not return to shore within the expected time frame.
He said there were crowds hanging around the beach waiting for quite a long time to find out what was going on.
When the announcement was made about the next three races being cancelled, everyone was understanding and left, Mr Pugh said.
"Everyone was pretty subdued, sitting around in groups and talking and hoping for the best.
"Not a single person complained. It's completely understandable the races were cancelled.
"Everyone just packed up and went home."
He said conditions in the water on Saturday were not the easiest, it was windy with quite a bit of chop. A swimmer could have passed within metres and not known someone was in trouble, he said.
"When you're out there, it's so rough and bumpy you're sort of looking after yourself. It would be hard to see anything around you. You can see the swimmer in front of you and that's about it."
Mount Maunganui Lifeguard Service said on Facebook that their "heart-felt condolences go out to the family of a swimmer who died at our Main Beach today".
"Incidents like these are incredibly traumatic for everyone involved, though the professionalism of lifeguards and other emergency services was again evident," the post read.
"At the same time as this incident was unfolding, our lifeguards found and reunited a lost 11-year-old girl with her worried family.
"Not all responses have a happy ending on the beach but rest assured, we'll give every one our best shot."
St John Ambulance confirmed two vehicles attended an emergency incident before noon where a person was given treatment but died at the scene.
NZ Ocean Swim Series event director Scott Rice confirmed Mr Smith had been part of the event.
"A person was rescued from the water in one of our swims. They brought them to the beach, first aid was performed and he was placed in an ambulance," he said.
"I know no more from there."
The competition's first event, the long swim, started at 9.45am and was a 2.6km swim around Moturiki (Leisure Island) and Motuotau (Rabbit Island). All other events of the day were cancelled following the swimmer's death.
