Fellow athlete Elisa Cusma put it most bluntly when in 2009, after coming sixth in a race that Semenya won, said, "These kind of people should not run with us. For me, she's not a woman. She's a man."
It highlights a widespread confusion about Semenya and her gender, with some confusing intersex traits with being transgender.
According to a leaked report from 2009, Semenya has a number of physiological features of a man. According to a leaked medical report from 2009, the runner has internal testes and a condition called hyperandrogenism, which naturally increases her levels of testosterone. This, in turn, can aid in the building of muscle - essential for an elite athlete.
Rivals say they would be labelled a drug cheat if they topped up their testosterone to the levels of Semenya.
In the past, that has been reason enough to exclude Semenya, and others like her, from the sport unless they took medication to reduce their hormone levels.
That rule has now been scrapped by the IAAF, the world athletics global body, under some duress - but some are still questioning Semenya's right to compete.
In July, British Olympian Paula Radcliffe told the BBC: "When we talk about it in terms of fully expecting no other result than Caster Semenya to win that 800m, then it's no longer sport.
"It's not just Caster's rights but all the women with elevated testosterone that need to be balanced with those that don't."
Radcliffe said Semenya was not at fault but suggested corrupt countries looking for an unfair advantage might scour "communities where the condition of intersex and hyperandrogenism is more prevalent", take them away and train them for future Olympics.
Nonsense, says Morgan Carpenter, the co-chair of Organisation Intersex International Australia (OII Australia), an advocacy group for people with intersex traits.
"This notion intersex people taking over the podium is ridiculous," Carpenter told news.com.au, noting that between only 0.05 and 1.7 per cent of people were born with these traits.
Increased levels of testosterone is one of more than 40 characteristics that stand outside the common binary definitions of male and female, which could indicate someone is intersex.
"There's an assumption being intersex you're not a woman and not a man, so we need to get a better understanding of intersex," said Carpenter.
For starters, it is not the same as being transgender. People who are intersex have not changed their gender.
According to the United Nations, intersex characteristics can range from the physical, such as the genitalia, to chromosome patterns with increased hormones common in the opposite gender. These might only be at very low levels, have no physical signs and leave someone completely unaware they were potentially intersex.
But some children born intersex (and who display it physically) have been forced to have invasive surgical and other procedures so their bodies conform to what society says is normal - often for no health benefits and with debilitating results.
This is particularly so for athletes. Carpenter said desperate attempts by some female intersex athletes to conform to gender stereotypes had involved awful operations including sterilisation, the removal of testicles and even operations to reduce clitoris size.
"That is actually female genital mutilation and the UN recently said those medical interventions are unnecessary so the consequences can be dramatic," he said.
It should be noted that Semenya has never confirmed her medical records or whether she is intersex.
In fact, she says the leaking of her medical details, and the constant scrutiny of her gender, has had a high emotional toll.
"I have been subjected to unwarranted and invasive scrutiny of the most intimate and private details of my being," she said in 2010.
For other intersex athletes, the consequences of having their genetics questioned have taken them to dark places.
Indian runner Santi Soundarajan, who failed a so-called "gender test" in 2006, attempted suicide the following year after she was stripped of her medals.
"Everyone was looking at me in this new way," she told sports network ESPN. "Is she a man? Is she a transvestite? It's very hurtful. It ruined my life and my family's life."
Carpenter said the focus on testosterone didn't extend to male athletes. Those who had low testosterone levels, "within a typical female range," were never questioned as to whether they were really a man. Even with her supposed advantages Semenya has never won an Olympic gold, only achieving silver in London 2012.
Elite athletes have a range of unique attributes, he said, that helped them win medals, such as Michael Phelps' enormous 203cm arm span, which was marvelled at rather than frowned upon.
"Every Olympic athlete has an advantage, they are all genetically unusual, there is no greater genetic advantage than any other," said Carpenter.
Indeed, a 2016 paper by the American Medical Association (AMA) poured cold water on the suggestion testosterone would make enough of a difference on its own to support exclude those athletes.
"Many factors, aside from sex and hormone milieu - favourable genetics, height, muscle type, economic opportunities, access to facilities, and skilled coaching among them - contribute to competitive success in sport."
The AMA said that a recommendation published 25 years ago, that women should be able to compete as women whatever their genetic make-up, was still relevant.
The research pointed out many competitors, often those that had lost against athletes such as Semenya, had commented on their competitors' "masculine" appearance and it was this that had led to the humiliation of them having to prove they were a woman.
Carpenter said that, more than any other factor, this could be the real reason these athletes get such a hard time.
"This is very much about how women are supposed to look. Caster Semenya was identified for sex testing because of the way she looked and the way she ran.
"She was publicly humiliated, paraded all around the world and her gender and the right to compete in the body she was born was brought into question."
But still the question persists, do genetic differences in some athletes lead to an unfair advantage when the big race is on?
Carpenter points to the IAAF's decision to suspend the rule barring athletes with higher testosterone levels from competing.
"It was suspended because there was no scientific evidence for a meaningful advantage," he said.
"Semenya competing is absolutely fair but it seems the court of public opinion has been called upon to judge her femininity."
