Mr Silverwood says the building's main attraction, the exhibition space, will charge for entry for information about Paparoa National Park which DOC is legally obliged to provide free of charge for everyone.
In their resignation letter to Conservation Minister Kiri Allan the three members say they have come under attack for raising the issue and are being prevented from carrying out their duties under the Conservation Act.
"We feel that board meetings and internal communications are no longer safe working environments... personal attacks, including claims of racism, have become normalised."
There had been a failure to meaningfully address conflicts of interest held by some board members, and the relationship between DOC and Poutini Ngāi Tahu was increasingly economically focused, the letter said.
This was at odds with conservation legislation that had no economic mandate, they said.
"As the DOC/iwi treaty partnership strengthens and develops, it is increasingly difficult for the board to remain independent of this co-management system, particularly as some board members are also the treaty partner representatives of their rūnanga."
The trio said DOC's western South Island director Mark Davies had failed to uphold the equal legitimacy of all board members.
"This has created ongoing difficult relationships between us and the director, which has restricted our ability to contribute to conservation outcomes for the region.
"We would like to suggest that the board is placed into statutory management while a review is undertaken by the Ombudsman or independent panel."
The aim should be to develop a board that has genuine treaty partner relationships on a conservation pathway, the trio said.
"We believe the issues between Ngāi Tahu and the West Coast Tai Poutini Conservation Board and the conflicts of interest that have gone unchecked at board meetings must be addressed."
Despite extensive efforts, the three members said they had been able to achieve little for conservation in their communities.
"The enthusiasm with which we accepted the public service appointments two years ago and the diligence and commitment that we have brought to the roles have been to no avail.
"As we become increasingly disenfranchised in advocating for conservation within a conservation board, it is tragically and abundantly clear that further efforts are futile and there is no other option open to us but to resign from our roles as board members."
Ngāi Tahu members of the board have rejected Mr Silverwood's claims that they are conflicted, saying they will make no personal financial gain from the visitor centre.
Ngāi Tahu board member and Makaawhio chief executive Kara Edwards said Mr Silverwood and his colleagues were completely missing the point.
"There would be no new visitor centre at Punakaiki without Ngāti Waewae," she said.
"The project is being funded from the Provincial Growth Fund, and one of the conditions of that fund is that the Crown cannot own the asset.
"Ngāti Waewae pitched the project with support from DOC and the lease arrangement will be the same as in many other places where DOC rents space for its visitor services."
The new gallery would offer visitor experiences above and beyond what DOC currently offered at Punakaiki, Ms Edwards said.
In a statement representing Poutini Ngāi Tahu and sent to media in the names of Francois Tumahai and Paul Madgwick, the group rejected the claims of the departing trio, saying the Conservation Act was required to give effect to the principles of the Treaty of Waitangi, and the iwi was actively working with DOC in partnership at a mana ki te mana level.
"We understand that the change to implementing a partnership approach is challenging for some individuals, however this is specifically mandated by section 4 of the Conservation Act... we look forward to working with new board members and implementing a treaty partnership in a modern context as soon as possible."
A spokesman for the minister said: "Conservation boards are independent, and it is for members of the board to comment on their internal matters". The minister has yet to respond to the resignation letter.
Correction: Paul Madgwick is the editor of The Greymouth Star, which employs the West Coast Local Democracy Reporter. This updated version of the story was edited by RNZ, with one change, to the third from bottom paragraph, to make it clear the Poutini Ngāi Tahu statement was sent out in the names of Francois Tumahai and Paul Madgwick.