The Hillary family wanted the house to be saved but could not afford to pay for a move.
Mountaineer Graeme Dingle offered to move it to a bush property he owned at Whakapirau, and let the Hillary family use it for holidays. But there would have been no public access.
Mr Jarvis eventually gave the house to the Sir Edmund Hillary Collegiate in Otara to be used as a leadership centre.
But more than four years on, no work has been done to develop his backyard.
He told the Herald he did not need the property and was more worried about someone else buying and building on the land and the possible effect on his own home.
He put the land on the market a couple of months ago after a handful of people approached him, wanting to buy the property.
But he did not have enough time to look at possible height restrictions then and was not actively marketing the property. He would do so again in the future.
"Someone else would get much better use out of the land."
Sir Ed's daughter Sarah Hillary, who grew up in the home with her brother Peter, told the Herald she did not know the land had been on the market.
Of the height restrictions, she said: "I suppose I can understand why he would do that, but it's going to affect how easy it is to sell it.
"I'm sure it will reduce the value, but that's up to him. I guess he thinks it's worth selling.
"It'd be interesting to know if all the old plants are there. I could go down and get some feijoas next season.
"It had a very good feijoa tree."
Jarvis' own four-bedroom stone home is 592 sq m and is on 2162sq m.
He bought it in 2008 for $8.5 million and its rateable valuation is $8 million.
Real estate agent Graham Wall, who marketed the property, described the area as an "extraordinary beautiful site" with views over King's School grounds to the harbour.
The price was by negotiation and the valuation - set in 2011 - is $1.46 million.
But that would not take into account any height restrictions.
Kensington Swan property lawyer Matthew Ockleston said restrictions were not uncommon but could greatly affect the sale price.
Vendors could either register a land covenant on the title with Land Information New Zealand before a sale, or they could include a caveat in the sale and purchase agreement.
"You put a covenant on that they only build a one-storey house or it's not pink or whatever it is you don't like. It's quite common with beachfront properties.
"People will sell off the front yard but they don't want to lose their beautiful view."