"They are arriving on Friday and will be there over the weekend.
"The date of the wedding was known eight months ago so it's strange Camilla organised her house party this weekend."
The alleged "snub" will fuel rumours that there is a royal rift at the heart of the monarchy. Camilla and Sarah Ferguson have never seen eye to eye because of Fergie's loyalty to Princess Diana, while there is also tension between Prince Charles and his younger brother Prince Andrew, often referred to as the Queen's favourite son.
A source close to the couple denied there was any ill-feeling, saying: "They get on very well, the couple has no issue with it - they have known about the diary clash for some time."
They added that Prince Charles and the Duchess of Cornwall "host events every day throughout this period every year."
It is believed the Duke and Duchess of York tried to persuade Camilla, 71, to attend but to no avail.
A source said: "It's hard to see her turning down the invitation as anything other than a snub but she insists she has other engagements that she was invited to first."
Camilla's no-show will be seen as opening up family rifts at a time when the royals are riding a wave of popularity following Harry and Meghan's wedding in May.
Prince Charles - 70 in November - and his younger brother Andrew have different visions for the future of "The Firm" after the Queen and Prince Philip have died. Andrew wants his daughters Beatrice and Eugenie to have greater roles but Charles believes the monarchy must be slimmed down to survive and show value for money.
The rest of the royal family - including Charles and 97-year-old Prince Philip - are expected to join the Queen at the 11am wedding at St George's Chapel and at a castle reception afterwards, although a royal source said Philip would make his mind up on the day. "He now very much operates on a "wake up and see how I feel" basis," said the source. The Duchess of York, who split from Andrew in 1992 but still lives under the same roof, is said to be "very nervous" about meeting Prince Philip, who has avoided her since her 1996 divorce.
"Sarah is already a bag of nerves and Camilla's decision won't help" said a friend.
"She is worried about the reception she will get from Prince Philip and now there will be a bit of a frosty atmosphere with Prince Charles."
Around 400 guests will then go on to an evening reception at Andrew's home nearby, Royal Lodge.
Eugenie, 28, and her fiancé Jack Brooksbank, 32, have invited 850 guests, compared to just 600 who attended Harry and Meghan's nuptials.
Guests at the wedding - which will be televised on ITV's This Morning - include the Clooneys, artist Tracey Emin and Robbie and Ayda Williams - although singers Elton John and Ed Sheeran have had to turn down their invitations because they are on tour.