Four refugees from Manawatu had addressed the Monday night gathering and Ward had been outlining Immigration NZ's policies and procedures for refugee settlement before Mair spoke.
Iwi members stood and sang a waiata at the conclusion of his speech.
Ward announced to the gathering that it was time for a dinner break before the second part of the hui which was to consist of workshops.
Mair later said that he had a "respectful discussion" with the hui organisers and it was agreed that the second part of the event would not proceed.
"Whanganui iwi did not receive invitations to the hui and it seemed like a further example of the process going on without us," Mair said.
The Whanganui District is expecting to get up to 110 refugees annually from March 2020.
Immigration Minister Iain Lees-Galloway made the announcement in February and said he was delivering the news well in advance so the city would have time to prepare.
Mair said a group of iwi representatives including himself and Dame Tariana Turia met with Whanganui Mayor Hamish McDouall after the announcement to voice their concerns and requested a meeting with the minister.
"He responded with an offer to meet in Wellington and we replied that we would like him to meet with us in Whanganui because this concerns our home."
The meeting has not eventuated and Mair said the hui at the Whanganui War Memorial Centre was the first meeting on refugee settlement he has heard about since the February announcement.
"This is not about iwi discriminating against refugees, we just don't feel we can welcome new people while our own are struggling.
"We are being asked to welcome 110 people a year but I can think of 120 local families living in overcrowded, unsuitable and unaffordable housing that need to be rehomed right now.
"Recent rent increases in Whanganui are appalling and people cannot afford $400 a week."
Immigration New Zealand's national manager of the refugee division Andrew Lockhart said INZ acknowledged that engagement with iwi is important.
"Iwi provide a valuable contribution to support refugee resettlement around the country.
"In addition, INZ understands the Minister of Immigration is also happy to meet with the iwi."