Started in 1876, the hospital treated servicemen from World War Two. It was modernised in the 1950s and 60s before closing in 1990.
The government bought the property from the Taranaki District Health Board in 1997 for 2.2 million. It was then sold to Mr Anderson at public tender for $25,000 in 2002.
Described by locals as "disgusting", it's been reported many are saddened the former community institution is now used to house pigs and other livestock.
Former nurse Sue Wilkinson is part of a group pushing to get the building demolished.
"The community is tired of having to look at this terrible sight, an embarrassment and a slant on our community."
"Its quite sad."
She's not surprised Mr Anderson didn't take the offer.
"The government will just have to come up with more money."
"We're not giving up," she said.
Mr Borrows acknowledges the government is partly to blame for the situation.
"I hope that the government learns from this, there's got to be some policies in place around what is done with land once its sold."
"It was believed the Mr Anderson would demolish the building, but from my understanding there was nothing in the contract."
"Its such a shame because the people of Patea are now left with this."
The quotable value of the entire property sits at $110,00 but Mr Anderson believes it's worth far more than what he's been offered.
"Six and a half acres, two houses, million dollar view you can see two mountains, see the sea."
"I'd be a mad person to take his offer," he says.
He thinks the value of the property may increase.
"I'm waiting for all the Christchurch people to come and live this way, or the Americans coming over. From what I see on the news most people won't be able to by a house soon."
"In 10 years time I know I'm going to get more than $150,000."
He previously told the Chronicle he would sell the property for $350,000 and although he bought the property cheap he has put in a lot of work over the years fixing the place up.
"I've got rid of four and a half tons of asbestos."
He says the attention has been hard on him and his family.
"I don't think its fair on my kids."
"I just want to live my life, be happy and tidy it up slowly."