"He's a self-righteous pig. He should try and come out and see what it is like," said one man, who has lived on the streets for more than 15 years.
"Hustling and begging is part of it to get by," said another rough sleeper.
Auckland City Mission's Chris Farrelly said: "We share a common vision here. Sir Bob wants people off the street. So do we."
The number of rough sleepers in the central city doubled to 200 between 2015 and 2016, he said.
In Auckland, passive begging is allowed but aggressive and intimidating begging is not under the council's public safety and nuisance bylaw.
Since the bylaw came into force in 2014, the council has taken the approach of providing advice and education in response to most complaints about aggressive behaviour.
Up to May last year there had been 16 prosecutions taken, of which all but one resulted in people being convicted and discharged. One person was fined $100 plus $130 court costs.
Bylaw and compliance manager Max Wilde said there had been a trend towards passive begging since 2014.
He said the bylaw was due to be reviewed next year where the governing body had the ability to make begging illegal. The review would include public consultation.
Farrelly said people begging on the street was not the Auckland or New Zealand people wanted and it had to stop before it was out of control.
He called on the multimillionaire businessman to help find houses for people who were genuinely homeless, saying 55 per cent of the 200 homeless surveyed in May last year were Maori. The other 45 per cent were Pasifika and non-Maori.
Farrelly said a two-year, $4 million-programme funded by the Government was under way this year to house the homeless and provide wraparound social services.
The Auckland City Mission and Life wise had joined up in central, west and south Auckland to commence the housing-first programme, he said.
Auckland Mayor Phil Goff is proposing to put $500,000 into this year's budget to coordinate the efforts of those involved in the homelessness problem. A spokeswoman for the mayor said he was not commenting on Sir Bob's views.
A homeless expert also agrees with Sir Bob that begging was a blight on society but disagreed on why New Zealand had the problem.
"We need to look at inequalities and growing poverty, and these things are getting worse and we are going to get more of it," said Massey University Professor of Psychology Darrin Hodgetts.
Blaming the people at the bottom was not going to address the causes, said Hodgetts, who has researched homelessness in New Zealand, the UK and Canada.
What the bylaw says:
• A person must not use a public place to: beg, in a manner that may intimidate or cause a nuisance to any person.
• The bylaw allows for passive begging, but not aggressive or intimidating behaviour.
• Council provides advice and education in the first instance. If this does not work it may take a prosecution.