Australia's memorable win soured veteran Springboks captain John Smit's 105th and last Test in his home city.
But it was the big away scalp the Wallabies craved before heading to New Zealand for the World Cup next month.
South Africa led 6-0 at halftime but, despite only six points being scored, it was a frenetic, action-packed opening 40 minutes.
Francois Steyn drilled a 48-metre penalty goal in the second minute after Elsom was booked on a touch judge's report for over zealousness before Butch James made it 6-0 in the 20th minute after the Australian scrum was pinged.
Aided by a lopsided penalty count, the Springboks enjoyed a territorial advantage in the first half but the Wallabies looked far more likely on attack in the rare moments they had the ball.
Fullback Kurtley Beale went within a few metres of completing an exciting long-range raid from the Wallabies after the halftime siren had sounded, but coach Robbie Deans was relieved to have his side in the sheds still in the contest at the interval.
The Boks could easily have been some way in front, with James missing an easy penalty goal attempt, then skewing a shot at drop goal, while the video referee took an age to deny a try to Jaque Fourie in the 25th minute.
Quade Cooper erred in letting a box kick from Boks halfback Fourie du Preez bounce close to the Australian in-goal but fortunately for the Wallabies the TV match official deemed the South African centre had knocked on.
It didn't take long for the Wallabies to hit the front for the first time after the break, with O'Connor slotting a penalty in the 44th minute before putting McCabe over for his first Test try shortly after.
O'Connor missed the conversion and the Springboks regained the lead through another James penalty in the 58th minute.
Some awesome defence from replacement centre Anthony Faingaa allowed Australia to nudge ahead 11-9 thanks to a 38-metre penalty goal from O'Connor.
Faingaa, not long on for Adam Ashley-Cooper, rushed up on JP Pietersen and forced the penalty when the Boks winger was unable to release the ball.
Despite driving rain lashing into his face, O'Connor calmly landed the difficult shot at goal.
Another O'Connor penalty six minutes from time gave Australia a little more breathing space, ensuring the Boks needed a converted try, and the Wallabies defended for their lives as the world champions attacked relentlessly in the tense final stages.
- AAP