A Hastings District Council spokeswoman said it had received numerous complaints in the past couple of weeks and had been trying to work with Mr Meier.
There were potentially a number of consent issues the council was working through.
Many of the complaints received had been about the fence, but as it was not attached to the ground and was referred to by Mr Meier as a work of art, the council was investigating what, if anything, it could do about it.
Other complaints referred to noise, which the council was working on, and that the property was "ugly", which the council said it could not do anything about.
One texter also described the scene as "ugly" and said the fences were "a mess".
Mr Meier, who has lived in Haumoana since 1994, said he had some simple but sincere words for people who had been attacking him verbally, as well as attacking his property. "It is a work in progress and is nowhere near finished," he said.
"I know it's controversial but people need to give it some time."
He said he wanted to do something colourful and interesting for the seaside district, where several businesses had closed over the past year. He said he bought the building to run an equipment-hire business but then had a change of heart.
"I thought ... let's open a cafe."
After he broached the idea on Facebook he got 400 "likes".
"So I thought, oh, I'll have to now ... I'm a builder but now I'm running a cafe." People who called by last weekend said they liked the unusual look.
"There's just some who don't but they have to give it time."
Mr Meier has also started bicycle rentals and in the long term is considering a beer garden. He has also started an art gallery inside. Mr Meier said his only issues were some of the threats he had received.
"One guy said he was going to burn the place down," he said, adding another had stopped and insulted one of his workmates there.
Local resident Shirley Hutchinson said she thought the cafe was a good idea until it started to develop, when it just got worse and worse. Now, she feels sorry for the people who live across the road from it.
She said Haumoana used to be a quaint little village but the new cafe and other distasteful properties meant the town was "going to the dogs".
"I think the council needs a wake-up call, because we don't seem to get any attention down here."
Some locals, however, offered support. One, after hearing about the downed fences, offered to take along his tools and give Mr Meier a hand.