Jesse Farry (26), a Dunedin teacher who was among the people told to leave the water, said the sea lion had been swimming near beach-goers when it started to become ''a bit aggressive''.
The behaviour continued after they were out of the water.
''It came up on to the beach and it was sort of charging at people, being a little bit aggressive, and opening its mouth and barking.
''Everyone just kept moving back and then it just sat down in the middle of the flags, plonked itself on the sand and sat there.''
The sea lion swam away after about five to 10 minutes.
''No-one was scared. Everyone thought it was just interesting.''
New Zealand Sea Lion Trust chairman Steve Broni said the sea lion probably wanted to play.
''His goal is not to bite anyone's leg off or anything, but sea lions do tend to play with their mouths.''
The worst possible outcome would likely be a playful bite, but sea lions' ''razor sharp'' teeth could cause some damage.
''To the best of my knowledge, no-one has been seriously injured by a sea lion on the Otago coast.''
Mr Broni said a lot of people made the mistake of trying to shoo sea lions away like they would with another animal.
''The more gesticulation you make around them, the more they take it as an invitation to play.''
According to the Department of Conservation, the sea lion is a 3-year-old male called Seazar.
It was pictured in Saturday's Otago Daily Times playing with a ball on St Kilda Beach.
On Saturday, a sea lion approached and barked at a swimmer in the surf at nearby Smaills Beach.
- ODT