"He hired a tent, he has got loads of friends, but he hired a tent just for the six of us so that we could sit outside. He stuck to the spirit and the letter of the rules. At that dinner, I get a call, my daughter has collapsed. As you will know, for months she had been isolated, she was ill.
"By the following morning, she had died and she had stuck to the spirit and letter of the rules. Now, there are going to be thousands of people who have that story in their background. And if I may say so, you are in here telling me about a civil servant's inquiry. That will not answer that anger," he added.
"Does the Prime Minister really understand why people are angry?"
Phillips says his story is one of thousands of people in the UK who stuck to the rules while Downing Street politicians partied and that no civil servant's inquiry could "answer that anger".
His daughter, freelance journalist Sushila Philips, died in April last year, aged 36, after living with anorexia for 22 years.
"I know how much you have suffered during this period," Dowden responded.
"What I'm trying to offer to you and to your viewers who may be watching this programme is to explain to you how the government is seeking to move forward through this," he added.
"First of all, to establish all of the facts of what happened. And then the Prime Minister will be held to account in Parliament for what happened."
Dowden said he was "disgusted" that the parties were taking place before the Duke of Edinburgh's funeral.
"We all shared and respected the tremendous dignity and selflessness that the Queen showed on this occasion as she has done throughout her reign," he said.
"I was disgusted at the fact that there were parties taking place before this profound event of national mourning in our nation's history.
"It shouldn't have happened and it was wrong, they've already apologised for doing so. It was just wrong and it shouldn't have happened."