Castle called the decision "a painful situation for the game" but said Folau left them with no choice.
"While Rugby Australia accepts the panel's decision directing termination of Israel Folau's playing contract for his high-level breach of the Code of Conduct, we want to stress that this outcome is a painful situation for the game.
"Rugby Australia did not choose to be in this situation, but Rugby Australia's position remains that Israel, through his actions, left us with no choice but to pursue this course of action.
"This has been an extremely challenging period for rugby and this issue has created an unwanted distraction in an important year for the sport and for the Wallabies team.
"But our clear message to all rugby fans today is that we need to stand by our values and the qualities of inclusion, passion, integrity, discipline, respect and teamwork.
"I've communicated directly with the players to make it clear that Rugby Australia fully supports their right to their own beliefs and nothing that has happened changes that.
"But when we are talking about inclusiveness in our game, we're talking about respecting differences as well. When we say rugby is a game for all, we mean it. People need to feel safe and welcomed in our game regardless of their gender, race, background, religion, or sexuality."
Castle also added that representing Australia on the rugby field is a privilege.
"Selection is dependent upon a player's ability to contribute to the Wallabies, and the game of rugby itself, in a positive and consistent manner both on and off the field. When players sign a contract with the Wallabies, they sign up to the values of the team and the sport.
"Israel is a great rugby player and we are disappointed and saddened by the fact that he will not see out his four-year commitment to the Wallabies and Waratahs."
The move also ends Folau's time with the New South Wales Waratah in Super Rugby.
Andrew Hore, New South Wales Rugby Union CEO, added: "This has been a difficult time for our sport, for our fans, our players.
"Israel has been a valuable member of the NSW Waratahs team since he started playing rugby.
"While NSWRU is disappointed to lose a player of Israel's calibre, rugby has a Code of Conduct and values that we must adhere to ensure that our game remains a game for all, no matter people's background or beliefs."
A disciplinary hearing had earlier nine days ago ruled the Wallabies fullback committed a "high-level breach" of Rugby Australia's code of conduct when he used social media to criticise gays.
Now, termination will be a hammer blow to Folau, who is now set to be left with nothing after reportedly previously knocking back a settlement offer of $1 million, a figure that RA has denied.
With the panel ruling to terminate Folau's contract, RA will not have to pay a cent more on his four-year, $4 million contract.
The Wallabies star described his fallout with the governing body as "challenging" and spoke of being tempted by the "opportunity" to rekindle his career with the NSW Waratahs and Wallabies during a Sydney church address.
"All the materialistic things I have been able to have over the last number of years are slowly being taken away from me.
"It's been really challenging but also it's been encouraging to myself to see what my God is actually doing."
It's understood Super Rugby's all-time leading try-scorer would have been allowed to resume playing again had he agreed to take down his latest controversial post.
"There have been many opportunities to potentially make the situation a little bit easier. I could go back and play the game, get everything back to the way it used to be," Folau said.
"The way Satan works is he offers you stuff that could look good to the eye and makes you feel comfortable, and if you follow that path all the worries and troubles will go away.
"(But) it is always the will of God that comes first."
The Wallabies star on Tuesday night took to his social media profiles on Instagram and Twitter to replace the previous image which showed him pointing to the sky while celebrating scoring a try for the Waratahs.
In its place the former NRL and AFL star has posted an image showing the No. 1 written in tiny words of "God" with the phrase "God First" at its centre.
The same image has also been posted as his new Twitter account photo.
Folau had earlier started the saga by warning sinners would be going to hell unless they repented with this Instagram post:
