But Greens co-leader Marama Davidson said now is the time for New Zealanders to have a conversation about removing statues that critics say no longer have a place in New Zealand.
"These statues have always been a tribute to colonial oppression – that hasn't changed."
But Peters is not happy the movement is gaining pace in New Zealand.
"A self-confident country would never succumb to obliterating symbols of their history, whether it be good or bad or simply gone out of fashion," he said.
But Davidson, whose Green Party helps makes up the New Zealand First/ Labour Government, completely disagrees.
"The history that we have all been subjected to, was in itself a rewrite to begin with; a history that undermined and left out voices, stories and achievements of many Māori."
The tone of Peters' press release, sent by New Zealand First and not through Peters' official government channels, is one not seen since the 2017 election campaign.
There are 99 days until New Zealand's election and Parliament will soon rise for the election campaign period.
Peters said that a country learns from its mistakes and triumphs: "Its people should have the knowledge and maturity to distinguish between the two".
"The idea that statues of Captain Cook, the greatest maritime explorer of his age, be pulled down because of the history that followed him is disgraceful.
He also lists "what if" hypothetical questions, including:
• If one doesn't approve of war we pull down our cenotaphs; • Should we demolish every school that once applied corporal punishment? • Should Gandhi's statue be thrown in the Wellington harbour because we don't agree? • Should knighthoods to the undeserving be post-humously withdrawn? • Do Māori now disown our mixed heritage?