It comes after the Duke of York was banned from the royal event at the 11th hour following an intervention from the future kings who feared "backlash" if he attended the service.
A source told the publication that Andrew was told to stay out of sight "for his own good" following the tense family talks and while he is thought to have challenged the ban, it was to no avail.
Andrew was also prevented from attending the Royal Ascot last week by Charles and William and as such is "enraged" at the pair.
Speaking to the Sun a source claimed, "Andrew is desperate to rebuild his life and wants his titles back," adding, "But he is fuming with Charles and William for stopping his plans."
"He doesn't want to talk to them. Battle lines have been drawn and this has put a cloud over Balmoral in the summer."
Prince Charles, Prince of Wales and Prince William, Duke of Cambridge attend the Order Of The Garter Service. Photo / Getty Images
Sources claim there are fears the duke's anger could lead to a family fallout during their Balmoral summer retreat with Andrew either choosing not to attend or to willingly cause friction between him, and the future kings.
A source claimed, "Andrew is doing all he can to make amends for the shame he brought on his family for being involved in such a scandal.
"He wants to make it up to the Queen which is why he is doing all he can to see her as much as possible.
"The rest of the family, apart from Her Majesty, are united in feeling that he should stay out of the limelight and keep quiet having left such a stain on the family."
