"The collective view was that Andrew was damaging the entire family," the unnamed insider revealed.
And another said:
"Charles first decided that Andrew must go.
"He felt he's just run out of road. So he discussed it with William in the days between Christmas and New Year and his son fully agreed with him. William and Charles then informed the Queen that they were of the same opinion."
Ultimately, it was the Queen's final decision but she told them she shared their view.
The Queen and Andrew had the final conversations on the matter on Thursday.
A third insider said William and Charles shared similar views on how to handle the situation, and they both "thought it needed to be resolved".
A statement was released from Buckingham Palace on Friday - Andrew was forced to relinquish his royal duties as he faces a lawsuit alleging he sexually assaulted an American woman when she was 17.
A statement from Buckingham Palace, released today, said: "With the Queen's approval and agreement, the Duke of York's military affiliations and Royal patronages have been returned to The Queen.
"The Duke of York will continue not to undertake any public duties and is defending this case as a private citizen."
The palace statement came after more than 150 navy and army veterans wrote to the Queen asking her to strip Andrew of all his military ranks and titles amid his continued legal troubles.
A federal judge in the United States earlier ruled that Virginia Roberts Giuffre's civil sex abuse lawsuit against Prince Andrew can proceed.
In a major blow for the royal, the judge denied "in all respects" the Duke of York's request to dismiss the case.
- Additional reporting AP
