"I really fired him after it became the number one show," Trump said of the agent.
The comments were an unusual start to the bipartisan breakfast. But they were not so unusual for a president who prides himself on putting on a good show and garnering good ratings.
He has taken Schwarzenegger to task in the past for low ratings and accused the former California governor of siding with his political opponent, Hillary Clinton.
Schwarzenegger fired back on Twitter with a video responding directly to the president's comments.
"Hey Donald, I have a great idea. Why don't we switch jobs," Schwarzenegger said. "You take over TV, because you're such an expert in ratings and I take over your job and Then people can finally sleep comfortably again."
Eventually, Trump seemed to return to his prepared remarks for the event.
He thanked his supporters for their prayers during the campaign and he recounted his trip to Dover Air Force Base to welcome home the remains of a US service member killed in the line of duty.
"He died in defense of our nation," Trump said of Chief Petty Officer William 'Ryan' Owens. "He gave his life in defense of our people. Our debt to him and our debt to his family is eternal and everlasting."
"We will never forget the men and women who wear the uniform," Trump added. "Our security has been earned with their sweat and blood and tears."
Trump spoke hours after using Twitter to comment on reports of a contentious phone call over the weekend with the Australian prime minister, which ended abruptly over the issue of refugees.
Trump was unhappy with a deal cut by the Obama administration to take in about 1200 refugees from the Middle East who had been held on an island off the coast of Australia.
"Believe me, when you hear about the tough phone calls I'm having, don't worry about it," Trump said. "They're tough. We have to be tough.
"We're taken advantage of by every nation in the world, virtually," he added.
Trump repeatedly returned to policy in his speech, defending an executive order that blocked refugees and immigrants from seven predominantly Muslim countries. He framed the move as an effort to protect the country's safety and freedoms, including religious freedom.
"We will develop a system to help ensure that those admitted to our country fully embrace our values," Trump said. "We want people to come into our nation, but we want people to love us and to love our values, not to hate us and to hate our values.
"We will be a safe country, we will be a free country and we will be a country where all citizens will be free to practice their beliefs," he added.
The president also declared that he would work to repeal the Johnson Amendment, which prohibits some tax-exempt groups from endorsing political candidates. And he pledged to protect religious freedom.
"I will get rid of and totally destroy the Johnson Amendment and allow our representatives of faith to speak freely and without fear of retribution," Trump said. "I want to express clearly today to the American people that my administration will do everything in its power to defend and protect religious liberty in our land."
