"Some people do it to be funny ... Every party gets damaged billboards. There are just some people in New Zealand that think it's okay to damage other people's property and have no self-control," he said.
Baker emphasised that he believed the New Conservative is a very central party that has the values of family in mind and that if people looked into their policies they will they are not as extreme as some have suggested.
On its social media page, the New Conservative wrote it won't let things, like its billboards being defaced with swastikas, get in the way of its party's agenda.
"In politics, we are the most dangerous as true believers who are secure in the knowledge that we are here to save New Zealand," it wrote.
"We have become a real threat, not just to the extreme left, but also to the establishment
"The far left exhibiting their hateful anti-semitism and fascism via Nazi symbolism has not stopped us.
"Career academics attacking conservative ideas of family, fairness, justice and freedom, have not stopped us.
"Far Left suppression of conservative voices by attacking employers, schools, family members, doxing, has not stopped us.
"Because your life, your family, your freedom and your nation is worth every sacrifice."
Their signs in Christchurch, Wellington and across Auckland have been vandalised.
The electoral commission has been contacted for comment.
