Many were angry after a march and speeches that reinforced activists' demands for an end to a "messy and genocidal history" of two centuries of European occupation. They call Australia Day "Invasion Day".
"It can become ugly, but only if the dominant society rejects outright our legitimate claim to continuing sovereignty and dominion over our lands, natural resources and the naturally occurring biodiversity," said Michael Anderson, the only surviving member of the group of Black Power activists who founded the embassy in 1972.
The embassy was launched under a beach umbrella to protest at the then-Liberal Government's refusal to introduce land rights, and was re-established in 1992 to demand broader concessions, including sovereignty.
Sovereignty was specifically excluded in the report last week of an expert panel established by Gillard to form proposals for constitutional recognition, which included acknowledgement of Aborigines and Torres Strait Islanders as the first Australians, recognition of their languages and cultures, and an end to racist clauses in the existing constitution.
On Wednesday Gillard announced Arnhem Land community leader Laurie Baymarrwangga as senior Australian of the Year and yesterday placed inclusion as a key element of the Government's policies to close the gap between indigenous and other Australians. But Aborigines still fall behind in every measure of well-being, from employment, income and housing to health and life expectancy.
Yesterday Social Justice Commissioner Mick Gooda said the embassy's aims were still important.
"While their efforts have helped to make self-determination the overriding factor in thinking about Aboriginal and Torres Strait Islander affairs, there is still a long way to go," he said.
But Abbott yesterday inflamed tensions by saying the embassy should go. "I think a lot has changed for the better since [the embassy was established], he said.
"I think the indigenous people of Australia can be very proud of the respect in which they are held by every Australian.
"I think a lot has changed since then, and I think it probably is time to move on from that."
Anderson, who had earlier claimed his life had been threatened before the anniversary, told protesters: "To hell with the Government and the courts in this country.
"You haven't got a high hope to take us on. We will force these issues. Too many of our families have suffered for some bastard to get in the road ...
"You've got 1000 people here peacefully protesting, and to make a statement about tearing down the embassy - it's just madness on the part of Tony Abbott.
"What he said amounts to inciting racial riots."
Protesters circled the nearby restaurant where Gillard had awarded medals to emergency services officers in a ceremony also attended by Abbott.
As the leaders sheltered inside and riot police rushed to the scene, they banged on the windows chanting "racist" and "shame".
After about 20 minutes, and protected by 50 police and bodyguards, Gillard and Abbott fled the building.
Meanwhile, in northern New South Wales, more heavy showers added to the flooding of the past few days, isolating homes, trapping people in cars and houses, and cutting roads.
Rivers between Grafton and Port Macquarie had yesterday either broken their banks or were in danger of flooding, and in Coffs Harbour emergency workers sandbagged businesses and homes as Coffs Creek rose.
Residents were warned to be ready to evacuate homes in Gladstone, Smithtown and North Macksville, where a king tide threatens to pump down the Nambucca River. Almost 2000 people were told to leave the Tweed River area on Wednesday.
