Frustration was only heightened as the royal couple had a Netflix film crew shadowing them at the time for the documentary Heart Of Invictus which follows the Duke and his event for wounded and injured service personnel and veterans.
Prince Harry sat down with US Today host, Hoda Kotb for an in-depth chat which quickly resulted in the upset of many royal insiders and experts after he insinuated the Queen needed "protecting" from people around her.
Speaking to Kotb, the Duke said "I'm just making sure she's protected and has got the right people around her."
He added, "Being able to see her in some element of privacy was nice, and [to have] a chance to get back to the UK for a couple of years, apart from those two times for my grandfather [Philip's] funeral and for unveiling a statue of my mum,"
The Duke also revealed he didn't know if he would be able to attend his grandmother's Platinum Jubilee but in a statement released on Friday, the royal couple confirmed they and their children would be in Britain for the event.
Following the couples announcement, sources close to the royal family told The Sun that Buckingham Palace has a plan in place to make sure the couple don't "exploit" any of the Queen's Platinum Jubilee events.
The source said "Senior courtiers believe that Netflix will see it as one big opportunity to exploit their mega-millions agreement with the couple.
"So a team of palace aides will be on standby to keep a very close eye on the crew, and act as minders if needs be."