Both of the men accused of killing the soldier had been monitored by the security services for years but were not deemed a threat to life.
It emerged that the Kenyan authorities said they warned the British security services about one of the men and his links to extremism, but claim they were ignored.
Cameron's visit to MI5's headquarters to praise spies before these details were publicly known, coupled with his decision to go on holiday, has been criticised in the aftermath of the worst terrorist atrocity on British soil since 7/7.
In other developments:
Theresa May, the Home Secretary, said the attack was not the work of a "lone wolf" as four other men were arrested by police in connection with the murder.
Extremist preachers could be barred from appearing on broadcasters such as the BBC under government plans to expand the powers of Ofcom, the watchdog.
Lord Michael Howard, the former Conservative Home Secretary, suggested Labour and the Tories join forces to push through a "snoopers' charter" against the wishes of the Liberal Democrats.
On Friday, the day after Drummer Rigby was murdered, Cameron announced that the Intelligence and Security Committee would carry out an independent investigation into what the intelligence services knew about the alleged killers.
After visiting the scene of the attack in Woolwich, Cameron paid a trip to security services headquarters. He thanked staff for their work on the investigation and other counter-terrorism successes.
However, a Westminster source suggested that it was a mistake to praise spies so soon after the tragedy and amid concerns about potential intelligence failings as it could give the impression that Cameron was prejudging the ISC outcome.
The ISC will receive a report into the murder of Drummer Rigby on Thursday and then will consider the terms of a future probe.
Michael Adebolajo, 28, had been known to the security services and police for a decade and was arrested six years after protests by extremists outside the Old Bailey. In November 2010 he was arrested in Kenya after being caught trying to travel to Somalia, allegedly to join the terrorist network al-Shabaab.
Kenyan authorities say they returned him to British intelligence officers, who failed to take their concerns seriously and instead tried to recruit him as a source, reports suggested. The second suspected killer, Michael Adebowale, 22, was also known to police and the intelligence services.
One senior Westminster source said: "I just wonder what he was going to thank them for. While they [the security services] did tremendously well during the Olympics 10 months ago, a number of pieces of information have come to light since then. The interesting question is how much David Cameron knew about the potential intelligence failings when he went to see them [MI5]."
Another source said Cameron did not "exonerate" the intelligence agencies during his visit.
The source said: "He wanted to thank them for the work they had done both on that particular operation - there had been a lot of people working around the clock - but also more generally.
"The idea that he was either blaming the security services or exonerating them when an investigation into what happened actually hasn't taken place isn't right."
Cameron is spending a week in a privately rented villa in Ibiza. John Mann, a Labour MP, said: "People expect him to be at his desk leading from the front. Does anyone think Margaret Thatcher or Tony Blair would disappear at a time like this?"
Meanwhile, extremist preachers could be barred from appearing on broadcasters such as the BBC under plans being considered by the Government.
At present, Ofcom, the broadcasting regulator, has power to intervene only after a broadcast has been made.
May is considering extending Ofcom's powers to allow it to stop broadcasts of radical preachers.
Her intervention comes after the BBC was criticised for broadcasting an interview with Anjem Choudary, a radical preacher and founder of a banned Islamist group.
However Greg Dyke, the former BBC director-general, said the BBC's editorial independence must be protected.
