The Coalition went to the election with the promise to change the act to ensure that speech found to be offensive or insulting was no longer defined as racial vilification.
At present the act outlaws actions in public that, based on race, colour, national or ethnic origin are likely to offend, insult, humiliate or intimidate a person or group of persons.
This includes posting offensive material on the internet, publishing racially offensive comments in the print media, making offensive speeches at a public rally, racially abusing people in public places, or making racially abusive comments at sporting events.
The act makes exceptions for comments made "reasonably and in good faith", including artistic works or performances, academic or scientific debate, fair and accurate reporting on matters of public interest, and fair comment.
In the year to last June almost a quarter of the more than 2000 complaints lodged with the Human Rights Commission were made under the act, 192 of them citing racial hatred - up almost 60 per cent on the previous year.
The commission's annual report said more than 40 per cent of racial hatred complaints related to material on the internet, a "significant increase" on the previous year.
The Executive Council of Australian Jewry's latest annual report on anti-Semitism showed similar trends, with the internet accounting for many of what was the second-highest level of incidents on record.
The number of reports of violence against Jews or Jewish facilities rose 21 per cent to 657. They cited vandalism or attacks on synagogues or other facilities, but the report said most involved threats and abuse.
Abusive and threatening emails, printed material, letters and phone calls increased 60 per cent over the previous year.
But Brandis, who believes the human rights pendulum has swung too far to the left and has been too narrowly defined, says he wants to "rebalance" the debate, and has drawn up terms of reference for a broad review by the Australian Law Reform Commission of statutory infringements of traditional rights and freedoms.
"The classic liberal democratic rights that in my view are the fundamental human rights have been almost pushed to the edge of the debate," he told the Australian.
"It is a very important part of my agenda to recentre that debate so that when people talk about rights, they talk about the great liberal democratic rights of freedom of expression, freedom of association, freedom of worship and freedom of the press ...
"You cannot have a situation in a liberal democracy in which the expression of an opinion is rendered unlawful because somebody else finds it offensive or insulting."
His planned amendments have been rejected by Labor, Race Discrimination Commissioner Tim Soutphommasane, and indigenous and ethnic organisations.
A statement from groups ranging from the National Congress of First Peoples to representatives of Arab, Jewish, Greeks, Armenian and Muslim communities said they opposed absolutely plans to remove or dilute protections against racial vilification.