Fraser has form in this regard. In 1997, when trying to get back into politics (she was briefly Labour MP for Balmain), she said: "I wish I could be as outspoken, I suppose, as Pauline Hanson [the controversial far right politician] and say, 'Look, I'm sick and tired of the immigrants that are coming into my country.'"
But, look, enough of Dawnie and Aussies who don't like immigrants, Australian men who treat women poorly and those who play the sanctimonious patriotism card. Like the Eskimo languages, which have plenty of words for "snow" but not so many for "patio furniture", that kind of Aussie has many words for "un-Australian" but not for "free-thinking".
Scott may have erred in making his withdrawal statement read like a politician speaking but saying nothing: "My decision has been taken as a result of an extremely busy playing schedule and other commitments, both personal and professional."
He might have done better not to bubble-wrap it. Last year he told it like it was, saying he wasn't overly concerned about Rio because it was only an exhibition: "Whether I win an Olympic medal or not is not going to define my career or change whether I've fulfilled my career.
"It's nothing I've ever aspired to do and I don't think I ever will. It's all about the four majors and I think that's the way it should stay for golf. To play an exhibition event down there ... in the middle of the majors season, I don't think any other athletes in their sport would do that."
Absolutely. Golf has no place in the Olympics; it is not the pinnacle of the sport. Same for tennis and football and - note to Dawn - the only reason those three sports are on the Olympic roster is the same money you are berating Scott about.
Kiwis might feel differently - Lydia Ko and Danny Lee - have a chance of medals but the fact is these will be weakened fields.
There is an argument for amateurs playing at the Olympics.
Some pros support Olympic golf as they feel, in time, the gold medal will become golf's foremost prize. Doubt it. For that to happen, 150 years of tradition and honour will have to be deflated in favour of an Olympic movement still recovering from the smell of corruption.
In Brazil, they have hacked a brand new golf course out of a nature reserve - as if Brazil isn't already losing enough vegetation with almost 6000sq km of Amazon rainforest cut down last year. Then there's the Zika virus, the political crisis, doubts over visitor security in a country ranked high globally for homicides and violent crime, and water pollution.
IOC president Thomas Bach said last month even though Rio's polluted bay had not been cleaned up 100 per cent, it had reached 60 per cent - and would have been zero had it not been for the Olympics. Hardly the stuff legends are made of ...
I don't blame Scott one bit. He's inspired who knows how many young Aussies to take up his sport and he is the first and only Australian to win golf's greatest single prize - the Masters.
Fraser and her ilk should stay home, scowling at the immigrants and hopefully taking a lead from the other kind of Aussie - the self-deprecating wags who wrote the first line of the alternative national anthem ("Australians all, let us ring Joyce for she is young and free ...") and who live in a country where anyone can be prime minister as long as they have a degree and are not black, Muslim nor gay.