A Rotorua Lakes Council spokeswoman said some of the Thursday Night Market stall operators were offended by the flag being on display and the market manager had a "quiet word" to Mr Dobbs about the complaints and how it might be a good idea to take it down.
But Mr Dobbs refused.
"The most copied, the most expensive and the most collected military in the world is Third Reich; we can't deny that. I have no sympathies with them myself of course but if the public come and ask for it, what do you do?. What's the worst thing you can say as a retailer? No haven't got any," he said.
"If enough people came in and said it was offensive, I probably would take it down. Of course there are people out there who will be offended and so there should be, but there should be a freedom of choice for me to do what I like, we don't live in North Korea or Syria."
Mr Dobbs said it didn't bother him if he ruffled some feathers and he could see a time when it wouldn't be hanging from his window.
"When I sell it. No one can make me take it down, freedom and choice - that's why we went to war, to get rid of oppression and tyranny. People [tourists and locals] don't even mention it, I have to ask them. I do a bit of market research, do you find it offensive? And they said, 'No, not at all.
"It's a $30 screenprinted modern copy, probably made in the last 10 years. I'm thinking about buying some more, and I can get more, they're still making them, they're making them in New Zealand. What does that tell you? You don't make stuff in New Zealand if no one is going to buy it."
Rotorua RSA museum curator Alan Bines served in the New Zealand Army and Navy.
"Personally, I think it is a disgusting kind of thing. The fact is we fought against them and certainly I don't agree with a swastika being displayed," the 85-year-old said.
"I think for him to display a German swastika out there, many ex-servicemen still around would have problems with it, it brings back sad memories."
New Zealand Jewish Council chairman Stephen Goodman said it was disappointing people were making a profit in this way.
"Unfortunately it is not illegal in New Zealand," Mr Goodman said.
"It is history and it is valid to study it but it really depends on the motivation. If it's to learn from the past to ensure we don't repeat the future then it is quite a valid subject to study, but the glorification of it is quite grotesque and most offensive."
However a university academic said it would be a mistake to take any message from a swastika flag on display, other than as an invitation to buy it.
Associate Professor of Philosophy at Auckland University, Tim Dare, said on standard interpretations of the right to freedom of expression he thought Rotorua Collectors Centre owner Walter Dobbs had a point.
"Perhaps the defining feature of humans is their capacity to form and assess opinions and ideas.
"We should be very careful about stopping that process at the outset by prohibiting the expression of ideas. And we should not assume that it would be harmless to suppress 'bad opinions' - it's important to have them expressed too, so we can respond to them and show that they are wrong," Mr Dare said.
"Of course we know people are offended by some ideas and symbols, but that is the price we pay for the greater value of freedom of expression.
"There may be an interesting wrinkle in this case - it's not really clear that the shop keeper is expressing an idea at all. It's not as if he's a homeowner flying a swastika to show he supports the Nazi cause. He's a shopkeeper who sells war memorabilia. It is probably a mistake to take any message from his display of the flag, other than an invitation to purchase it."
