Some blame the location, 20km west of central Sydney - colonial buildings in the CBD, such as the Hyde Park Barracks, are impeccably preserved. Others note the associations with welfare and mental health - not popular drawcards. The Female Factory later housed a "lunatic asylum". And still others point to a modern- day ambivalence about Australia's early European history.
"I think there's a level at which we are ashamed of our history, we're not whole-heartedly committed to the idea that colonial history is something to be proud of and be preserved," says Elizabeth Farrelly, a Sydney-based writer and architecture critic.
While the Parramatta site is suffering "demolition by neglect", as Djuric puts it, heritage buildings in central Sydney are threatened by the march of developers. The New South Wales Government wants to sell two colonial sandstone buildings by the harbour which have never been in private hands. It also wants to sell former maritime workers' cottages in Millers Point, a working-class area undergoing gentrification. Many of the Victorian and Georgian terraces are occupied by public housing tenants.
In Windsor, one of five towns planned by Governor Lachlan Macquarie in 1810 on the Hawkesbury River, north of Sydney, a historic bridge is to be knocked down for a four-lane highway which will overshadow beautiful Thompson Square, the first town square built in Australia.
John McInerney, a former Sydney independent councillor and town planner, believes the picture is replicated across the nation, especially in Liberal-run states. "It seems each time a Liberal government comes in, the heritage controls are diminished and the development lobby becomes dominant," he says.
But it's not only Australia's built heritage that is menaced by growth and industrial development. In Western Australia's Pilbara region, 30,000-year-old rock art was destroyed by Woodside Petroleum's huge liquified natural gas project. North at James Price Point, another Woodside project threatens the raw beauty of the Kimberley coastline.
Unesco has expressed grave concern about the impact on the Great Barrier Reef and marine life of intensive port development. In Tasmania, construction of a four-lane highway near Hobart went ahead despite the discovery beneath a floodplain into which concrete pylons were sunk of a vast trove of Aboriginal artefacts from more than 40,000 years ago.
Tasmania is updating its laws to better protect indigenous heritage, but Aboriginal groups say the reforms are too weak. The NSW Government has foreshadowed new planning laws which the Greens claim will turn the state into a "developer's paradise".
Other critics say the Heritage Council, the statutory watchdog, will become a "toothless tiger".
While the trend in the 1970s to early 2000s was towards stronger heritage protection, says McInerney, the tide has turned, and "there's a new environment of economic rationalism, growth and profit at any cost". Farrelly points to changing aesthetics. "The fashion is neo-modernism in office and apartments and houses. So the heritage look is out of fashion, unless it's funky industrial chic."
The Parramatta River site is not even on the state heritage list, despite being "one of the birthplaces of our Australian nation", as Geoff Lee, the state Liberal MP for the area, puts it.
The state Government announced a "master plan" to renovate the 7ha site, but as Borger notes: "It's the third master plan in the last 20 years. Governments choke when it comes to spending the money."
Djuric now leads a state-funded "memory project", comprising artists, social historians and former occupants of the various institutions which have existed on the site. She wants memories preserved in an interpretative centre, the grounds "turned into a children's garden - a safe play area".
In 2009, Kevin Rudd apologised to the "Forgotten Australians", children abused in institutional care. But, says Djuric, "Nothing has really changed. This is a really big part of Australian history; why isn't it remembered?"