Her complaints sparked immediate action from the Albanese government.
Government Services Minister Bill Shorten has said he will scrap the gender inclusive forms and cited an interview Grover did with the Daily Telegraph when explaining his decision.
Shorten tweeted on Thursday morning he had read the newspaper story that mentioned a "pilot programme launched in three hospitals under the previous coalition government".
"When I was informed of this situation yesterday, I instructed the responsible officials they should cease using the previous government's forms," Shorten wrote.
"They will be replaced with new forms that use the word mother, not birthing parent, which is consistent with other Medicare forms."
Advocates say that using gender neutral language such as "birthing parent" can help include people who would otherwise be excluded.
Each year in Australia, several men give birth to children. In 2014-2015, as many as 55 men gave birth, according to Medicare records.
That's possible because not everyone's gender identity matches the reproductive organs they were born with.
The federal government in 2013 amended the Sex Discrimination Act 1984 to introduce new protections from discrimination on the grounds of sexual orientation, gender identity, and intersex status.
The change meant that government records should reflect the fact that some people identify with a different gender than the one they were assigned at birth.