He accused the current government of wasting $26 million in a "last-minute sales pitch that cheapens the issue" and treated the flag like a "supermarket special".
"Even if you are for the flag change, surely you can see that allowing anyone to vote is not right."
Change the New Zealand Flag chairman Lewis Holden shrugged off Mr Peters' speech and simply put it down to the New Zealand First Leader being "rattled by the growing support for a contemporary, modern flag to replace a tired old design that belonged to a colonial outpost of the British Empire".
Mr Holden said he hoped Mr Peters would have grown tired of playing the immigration card.
"Some people who have residency may have lived in New Zealand for very long periods of time and have legitimate reasons to keep the passport of another country," Mr Holden said.
New Zealand was a multicultural nation and the current flag was outdated, he said.
Act Party deputy leader Kenneth Wang, speaking to Mai Chen last week, said it was important for migrants to vote in the flag referendum.
"Express your view. Don't be silent to be seen, or heard."
He designed a flag and submitted it to the panel last year.
Mr Peters' call for immigrants to be blocked from voting in the flag referendum met with short shift from comedian Urzila Carlson, a South African who moved to New Zealand in 2006.
Although she did become a citizen in 2012, she said people with residency should be able to vote on the flag. "Yeah man, totes. We choose to live here and we are invested financially and emotionally in this country. They choose to live here and have to live and work under the flag! Just try and stop us from voting and having a say!"
The flag referendum opens on Thursday, March 3.