"As the couple made clear a few weeks ago, they are taking some time away together as a family," said one.
"This means that they will miss Christmas events."
Harry and Meghan are currently on a six-week break from royal duties to spend time with baby Archie and will be spending Christmas with the duchess's mother, Doria Ragland, at an undisclosed location abroad.
But some royal officials hoped Harry and Meghan might return long enough to join the Queen's huge staff party on Monday, which they have attended previously.
The Queen hosts the event out of her own pocket each year to thank her royal workers.
There was also talk that they would attend the annual turkey and trimmings lunch at Buckingham Palace on the Wednesday before Christmas, shortly before the Queen leaves for Sandringham for the festive season.
The cherished tradition is one of the rare times all of the monarch's extended family meet, and is a chance for the Queen to celebrate with those who do not receive an invitation to spend Christmas in Norfolk with the senior royals.
It is even hoped that Prince Philip, 98, will make the journey down from Norfolk, were he spends most of his time nowadays, to attend.
Family members are seated at round tables, enjoy a Sandringham-bred turkey lunch and pull crackers.
Seven-month-old Archie will also miss out on some fun with cousins Prince George, Princess Charlotte and Prince Louis, who he has barely seen since his birth, as children and babies enjoy their own Christmas dinner in a separate room.
Last month the palace confirmed Harry and Meghan's plans in a statement, saying: "Having spent the last two Christmases at Sandringham, Their Royal Highnesses will spend the holiday this year, as a new family, with the Duchess' mother Doria."
The couple have not been seen in public together since Remembrance Sunday, although Meghan has been posting fairly regularly on the couple's Instagram account.
They will return to the UK, possibly as late as early January, to launch their new charity after splitting from the Duke and Duchess of Cambridge.