As Senator Brandis walked off the stage at the Museum of Contemporary Arts, Mr Hicks told reporters he was a coward for not answering his question.
"He's run away," he said.
The incident came a day after a US Senate Intelligence Committee released a scathing report into the CIA's torture of al-Qaeda suspects.
The report said the controversial program was far more brutal than acknowledged and did not produce useful intelligence.
Adelaide-born Mr Hicks was 26 when he was captured by the anti-Taleban Northern Alliance in Afghanistan in late 2001. It believed he was fighting for Osama bin Laden's al-Qaeda network.
Mr Hicks was held in the US-run jail in Cuba until 2007, when he pleaded guilty to providing material support for terrorism and was sent to Adelaide's Yatala Prison to serve the rest of his seven-year sentence. He was released under a control order later that year.
He claims he was beaten, sexually abused and drugged while in Guantanamo Bay, and that he was convicted on a statement of facts for which he never received any evidence.
His father said David and his supporters have long suspected that Australia's government at the time, led by John Howard, must have known of his mistreatment.
"A couple of times when we visited David in Guantanamo he told us straight out that he was being tortured - not only him but the others," Terry Hicks told AAP.
"There was water boarding, there was the turning of the aircon flat out (to cold) and then reversing it back down to reverse heat, the noise and sleep deprivation."
The heckling incident overshadowed a number of announcements by Senator Brandis.
He declared earlier that all asylum seeker children would be released from Christmas Island by early 2015.
"Between now and Christmas, in other words, in the next two to three weeks, all of the children in detention on Christmas Island will be released from Christmas Island and will be returned to the community as soon as possible," he said to the applause of the packed crowd.
"All the children in detention will be released by the early months of next year."
He said the government would also ask the Human Rights Commission to conduct an inquiry into employment discrimination against older Australians and people with disabilities.
- AAP