Eventually, four attendees were asked to leave the venue, and two other men followed them soon after.
The play resumed after the delay with Australia reaching tea at 312-6 before declaring, setting India 407 to win.
On Saturday evening, India reportedly lodged a complaint to the ICC after two of players allegedly faced racial abuse from the SCG crowd on day three.
Livid Indian officials spoke with the ICC match officials and the stadium security after play on day three to make the accusations.
The Daily Telegraph reported the pair said they had been abused by members of the public for two days and claim it was racist in nature.
According to sources within the BCCI, Siraj was allegedly referred to as a "monkey" by a drunk spectator, reliving a similar incident on India's tour of Australia in 2007-08.
In commentary for Fox Sports early on day four, former Australian star Michael Hussey said: "It's terrible behaviour and I can't believe it's still happening in this day and age.
"They should be banned for life from coming to the cricket.
"The Indians have come here to our shores to entertain us, play some great cricket, we should be so grateful that we can watch some live sport. To treat the players like that is unacceptable."
Shane Warne added: "Disgraceful to be honest, absolutely disgraceful.
"Should never happen, especially with what's happened in the last 12 months or so with everything else around the world. Let's hope they come down heavy and find the culprits."