The protests were led by Māori community leader Dame Rangimarie Naida Glavish, who says the site is culturally significant and has nothing to do with the Erebus disaster.
However, mana whenua Ngāti Whātua Ōrākei had expressed its full support for the memorial and detailed its extensive involvement in the process.
Site preparation began in October.
David Ling, whose mother died in the crash, said he had experienced 40 years of difficulty and trauma.
He said the protesters had exacerbated that, and their actions might further hurt the families.
"One of the things I will be thinking about is I dread what antics they may get up to further upset the families," he said
Ling said his mother used to visit the gardens.
Air New Zealand flight 901 crashed into Mt Erebus in Antarctica on November 28, 1979.
All 257 people on board were killed, leaving their families and a nation bereft.
It remains the worst civil accident in New Zealand's history.