The wife frowned and the daughter moved closer to the middle of the boat, away from me.
I apologised again, untangling myself from their anchor rope and pushed off their hull to float away.
It's fair to say, at this point I wasn't completely enjoying my first experience of stand-up paddleboarding (SUP), the aquatic craze that has swept the world over the past few years.
The sport, which originated in Hawaii, took off in the mainland United States in the mid-2000s and has grown exponentially since.
It should have been perfect.
A stunning day - in Manly - with beautiful vistas all around. Boats docked in the harbour, ferries chugging by, kayakers carving through the water. But I felt like a polar bear on roller skates. Balance was hard to come by, and direction, well forget about it.
Fortunately my instructor Craig, who is one of SUP's pioneers in Australia, had seen it all before.
"Mate, the wind doesn't make it that easy today," he said reassuringly. "But you'll get there, everybody does."
Earlier he had taken us through drills and instructions on the sand, with some key tips: "legs relaxed, feet anchored, power from your core, look straight ahead" before we hit the water.
For the first 20 minutes, there was plenty of paddling; not much standing up.
I didn't have much control over where I was headed and boats, kayaks and other boarders became hazards.
But, as with most new activities, there was a breakthrough and then I almost instantly understood the appeal of the sport.
I had been a bit sceptical about SUP but I soon realised it's challenging, stimulating and, as you cruise along surveying the shore, it also feels pretty darn cool.
Craig taught us basic strokes and turns, correct stances on the board, what to do if there is too much wind and how to pivot quickly.
Suddenly the thought of paddling to a nearby bay or trying to catch a wave at Manly beach a few blocks away, seemed feasible.
Maybe I would even try SUP Ball next time, the sport invented by Craig's company, which is marketed as a combination of netball, water polo, lacrosse and rugby. Maybe.
CHECKLIST
Details: For info on stand-up paddleboarding, visit the Manly Kayak Centre.
The writer visited Sydney courtesy of Destination NSW.