He reported the latest incident to police who advised him to install a camera to get footage of the perpetrators.
"We're ultimately really positive people, and we've worked really hard on our business. We just want to make sure that the street is a good, positive place," Mr Brown said.
He has been in touch with City Safe, which said it will have extra patrols in the area this week.
Mr Brown welcomed the move, but is still considering sleeping overnight in the cafe on Friday in order to snap the perpetrators.
He plans to install a sensor light to help as a deterrent, and is calling for cameras to be installed on Walton St, to help identify the perpetrators and keep the street safer from vandals.
Mr Brown is grateful to the Design Centre design school next to his cafe, which is designing and gifting him a canvas banner for his front window.
Neighbouring business owner Qi Han Huang, of Absolute Roasts, has also been hit by the taggers.
He wants cameras installed on the street, and hourly patrols.
His shop is open until 9.30pm on Friday nights, and he often stays much later, sometimes sleeping there overnight - which is how Matt Brown knew the tags were made on Friday, not Saturday, nights.
Mr Huang, originally from China, said two weeks ago he woke to the sounds of activity on the street about 1.30am on Saturday. He was able to flick on the shop light, but didn't see who was responsible.
He is also frustrated at years of vandalism on the street.
Four years ago, the front window of his shop was smashed, costing him $2000 to replace.