"Those like Ngapari Nui are the ones that make rehabilitation a reality. Throwing away the last 10 years of hard and dedicated work is just a huge slap in the face," Ms Ngarewa-Packer said.
Dame Tariana Turia, a Ngati Apa/Whanganui leader and former leader of the Maori Party, also criticised the decision to remove Mr Nui. She said Mrs Collins had acted in a "high-handed manner" by bowing to "the prejudice of a pressure group".
"We are disappointed, but not surprised, that the Minister has elected to take her advice on the issue of iwi representation from a political advocacy group, rather than iwi leadership," said Dame Tariana.
"Kaiwhakamana are not employees of [the] department. The use of the term kaiwhakamana by the department is intended to acknowledge the role of kaumatua who fulfill the role of specified visitors to prisons."
Dame Tariana said the appointment of kaiwhakamana must be approved after by nominated by iwi or other Maori organisations.
She said there had been no complaints about Mr Nui or his behaviour at Whanganui Prison.
"As far as we are concerned, the support of people in the kaiwhakamana role are vital to build that trust relationship which assists inmates in taking a fresh look at their lives, and committing to a future in which criminal offending plays no part."
Meanwhile, Whanganui MP Chester Borrows said he had known Mr Nui for a long time and thought he would be an asset to the prison.
But he said the matter was over to the Minister.
Maori Party co-leaders Marama Fox and Te Ururoa Flavell also expressed disappointment at the situation.
"Mr Nui has a record of doing excellent work volunteering his time to help rehabilitate and reintegrate prisoners to the community. He's been doing this for years and there hasn't been a problem," Mr Flavell said.
"It seems some people have oversimplified this situation and are preying on people's fears and prejudice."