The battle for George: Prince Charles v Carol Middleton

Camilla, Prince Charles, Prince George,Prince William, Catherine, Queen Elizabeth II and Prince Harry look out on the balcony of Buckingham Palace. Photo / Getty Images

Sharing a joke with the Royals at Ascot or taking centre stage aboard a steamer for the Queen's Diamond Jubilee pageant on the Thames, there was a time when the Middletons were pretty much inescapable.

There could be no mistaking the message: this ordinary - if rather affluent - family was a new and most welcome addition to the Windsor family circle, and to national life.

Away from the television cameras, there were private deer stalking weekends at Birkhall in Scotland and pheasant shooting trips with Prince Philip on the Sandringham estate in Norfolk.

No wonder the Middletons, and Kate's mother Carole in particular, seemed to radiate high-wattage happiness.

Yet what a turnaround there has been. Not for the Middletons a place at Ascot this year, nor a pew in St Paul's Cathedral for the thanksgiving service for the Queen's 90th birthday, nor even a seat at the Royal Windsor Horse Show.

Today, in fact, their public appearances are few and far between, limited to events such as Wimbledon, which they attended as guests not of the Royals but of the All England Club.

Most significant of all, when Charles plays host to Kate, William and his two young grandchildren at Birkhall, his home on the Balmoral estate in a few weeks' time, Kate's parents will be decidedly elsewhere.

Amid the many theories circulating among courtiers as to why the Middletons have gone so inexplicably missing, there is one that is particularly intriguing.

For what is undoubtedly true is that Prince Charles believes he has been edged out by Carole in the battle for the affections of their two grandchildren - Prince George, three, and one-year-old Princess Charlotte.

As a friend puts it: "Charles feels very much that the Middletons get more than their fair share of time with George and Charlotte.

"In comparison, he sees them very little and he bears a bit of a grudge about that."

It can hardly help that courtiers feel Carole exudes a rather grandiose presence around Kensington Palace and is possessive of William, Kate and the children.

Indeed, since the birth of George, Charles has reportedly said: "They never let me see my grandson."

Is this the reason why relations between the families, which were warm and cordial around the time of the Royal Wedding in 2011, now seem more distant?

Take the matter of Birkhall, the Scottish estate that had belonged to Queen Elizabeth, the late Queen Mother.

It is only a few years since Charles generously lent William and the visiting Middletons the use of his ghillies so that they could enjoy a shooting weekend, and Carole, 61, enjoyed a weekend there as recently as 2014.

"But at the moment, there's no talk of them going," says a member of Charles's circle.

"He wants to see his son and daughter-in-law and spend time with his grandchildren alone.

"He has always wanted to mentor George, who is, after all, the heir to the throne, and sees it very much as his role.

"He will always insist the grandchildren come to Scotland so that they can learn to shoot, hunt and fish.

"That's very important to Charles. He wishes the family would do more traditional things like holiday in Scotland."

Charles did attend Prince George's third birthday tea party two weekends ago, but a source said it was very much on the Middletons' terms.

Catherine and Prince William with their children, Charlotte and George. Photo / Getty Images

"Carole organised everything and Charles and Camilla were, of course, invited," said the friend.

"He went, because he wasn't prepared to miss out but Camilla didn't go. Apparently she had other commitments."

It was the Middletons who were behind the scenes helping to organise the children's food and games, while Prince Charles had spent the morning at Clarence House in London meeting with manufacturer and designer Emma Bridgewater to discuss a campaign to protect rural England.

"Charles didn't want to miss his grandson's birthday but the fact that the whole thing was essentially a Middleton event with Carole orchestrating the whole thing is exactly what upsets him," said the source.

A family friend said: "Charles feels rather left out. He gets very little time with his grandchildren and I know he gets upset about it because he has said so.

"He certainly feels William spends more time with the Middletons than he does with his own family."

Another Royal source added: "While Charles would never dream of saying anything to upset William, he might be making a point by keeping the Middletons at arm's length."

A spokesman for the Palace declined to comment on any souring of relations, but there is no questioning the fact that the Middletons have dropped out of public view.

There was no sighting of Carole or Michael, 67, when the Royal Family gathered for the Queen's 90th birthday in June, despite speculation they might make an appearance at one of the celebrations which included a special horse show at Windsor and an historic lunch in The Mall.

And at Ascot last month they were nowhere to be seen, even though William and Kate attended the race meeting for the very first time.

It was a stark contrast to the Queen's Diamond Jubilee celebrations in 2012 when the Middletons were invited by the Queen to board the Elizabethan paddle steamer for the flotilla on the Thames.

Several weeks later, they were at Ascot, again at the Queen's personal invitation.

Carole and Michael, who own a stake in a racehorse and are both racing enthusiasts, had been extended the honour of riding in one of Her Majesty's carriages on Ladies' Day in June.

They travelled in one of only three landaus that followed the Queen's carriage after attending lunch at Windsor Castle with a gathering of the monarch's friends. This year, though, it seems the invitations have dried up.

The Royal source said: "I don't believe the Middletons were on the Queen's personal list of invitations this year, and it was unusual for them not to be there because they love racing.

"One gets the impression that perhaps a bit of distance is being enforced."

The Windsors and the Middletons wouldn't be the first new family partnership to find adjustment tricky - particularly those families lucky enough to have two sets of doting grandparents.

And it is bound to cause tension over who gets to see the children most.

William, of course, has always been fond of the Middletons and their straightforward nuclear family, something that the death of his mother had left him very much without.

And with the arrival of Prince George in July 2013, their influence, and that of Carole in particular, seemed to grow still further.

Kate and William seemed to prefer spending time with the Middletons rather than Charles and decamped to their home in Bucklebury in the days after George's birth.

Then, when they moved to Anmer Hall on the Sandringham estate, Carole became a regular visitor.

When their home at Kensington Palace was refurbished, William and Kate created a "granny annex" so Carole would always have somewhere to stay.

When the Duke and Duchess of Cambridge are overseas or on engagements, Carole helps the family's nanny Maria Borrallo care for Charlotte and George, and when the couple visited India and Bhutan earlier this year, it was Carole who took charge back at home.

There is no doubting that Charles would like to be a doting grandfather, given the chance.

He has, for example, gone to the trouble of refurbishing the thatched tree house in the shady section of the Highgrove garden known as the Stumpery.

It was there in the playhouse that Charles proudly staged a tea party for George when his grandson paid a visit.

It was his first such visit, and Charles planted a tree in view of the house in honour of the occasion.

Then, over in the wildflower meadow, Charles has installed a £20,000 shepherd's hut - complete with a little bed, woodburner and French oak wooden floors, again for the benefit of his grandchildren.

Yet the playhouse and the hut have been sadly empty - because the children's visits to Highgrove and its gardens are few and far between. Of course, it could be argued that Charles is in part to blame.

The source continues: "Charles is his own worst enemy. When he is at Sandringham, he doesn't want to bother William and Catherine, he waits for them to call rather than take the initiative himself.

"Occasionally, the couple come to Highgrove, but it's never relaxed unless Charles is in the garden with the children.

"The visits are occasional. It's a long way for the Cambridges to come from Anmer, and they've only been to Highgrove with the kids a couple of times."

Moreover, Charles is something of a workaholic, devoted to his charitable causes and official duties.

The Middletons, in contrast, have the freedom to run their own schedules and spend more time with their grandchildren.

A friend of Charles says while William and Kate enjoy holidays with the Middletons, it has been "years" since William has been away with his father.

"Charles would love to do something like a family skiing holiday, but William's not interested.

"He wants to do his own thing these days and often his father doesn't know where he is or what he's doing.

"When William and the family nipped off to Courcheval for a few days' skiing at the start of the year, Charles had no idea they were away. He was rather hurt.

"Neither of them pick up the phone to each other and weeks can pass without them having a proper catch-up."

The Middletons were last seen in the presence of the Queen in January at a service at St Mary Magdalene Church on the Sandringham estate to mark the 100th anniversary of the end of the Gallipoli campaign.

Carole, Michael, Pippa and James had been celebrating Kate's 34th birthday with the Cambridges at Anmer Hall and, because William and Kate were expected at the service, the Middletons were also invited.

It was the first time they had been seen with the Royal Family for some time and their presence prompted some commentators to question why they were not at the Queen's service of thanksgiving at St Paul's to celebrate her 90th birthday in June.

Then again, it might well be that the apparent distance is a sign of discretion on the part of the Middletons, anxious not to cause offence.

A friend of the Middletons says Carole, who joined the Cambridges on their skiing trip this year, was horrified to learn last year that Charles had privately complained to friends that he "almost never sees Prince George".

"She was devastated," said a friend.

"All she has ever wanted is to be a good mother and grandmother, not upset anyone in the process. Perhaps, their low profile is their way of keeping a respectful distance."