The beggars were all given trespass notices.
The cafe owner, who wanted to be known only as Ms Li, said those who were trespassed were banned from entering Esquires, but that did not stop them loitering around the outdoor seating area where smoking customers were, because it was outside the cafe.
It is very frightening because [customers] are not sure what they are going to do.
"This is very annoying to the customers, and it is very frightening because they are not sure what they are going to do," Ms Li said.
"The gang only focused on Asian customers. I think they have seen that they had luxury bags, late-model iPhones ..."
Last month, the group started throwing ashtrays and containers from tables after staff discouraged customers from giving them anything. One male staff member was also attacked.
Ms Li said the three police officers who came to the cafe around 4pm on Friday spoke to the five people, and told her all of them were given trespass notices.
Auckland's Asian communities have been on edge following a spate of attacks.
A 22-year-old business student from China, who gave her name only as Nina, said her iPhone 6s was stolen about two weeks ago in Freyberg Square - near the Esquires cafe - by an apparently homeless person.
"I loaned him my phone because he said he needed to make an emergency call because his friend was having a heart attack," she said. "But he took it and just ran away."
She did not make a police report because she didn't think they would do anything, but said the incident left her frightened.
Goro Usui, who owns Sushi Train Restaurant at Chancery Square, said his restaurants had also been hit by rough-sleepers.
He alleged that sushi, drinks and a tip box had been stolen, and toilets were periodically soiled by beggars.
"I have called the police more than 10 times, but most of the time, by the time the police get here, all the people are gone," Mr Usui said.
Next door, the manager of Courthouse Lane Grocer, Manvir Singh, said theft and abuse from homeless people was a regular occurrence.
Police Asian liaison officer Jessica Phuang urged all victims to make a police report, even if they thought the crime committed was small.
"The worst thing a police [officer] would say is that, I am sorry, we cannot investigate this case."