A controversial paper by a Canadian professor published earlier this month in the Harvard Public Health Review called for the Games to be cancelled or moved because it said they would likely speed up the spread of Zika throughout the world. Several health experts have disputed the report as lacking evidence for such a move.
"The risk to delegations going and athletes is not zero, but the risk of any travel isn't zero. The risk is not particularly high other than for pregnant women," Dr Frieden said.
Zika infection in pregnant women has been shown to be a cause of the birth defect microcephaly and other serious brain abnormalities in babies.
The World Health Organization has also said there is strong scientific consensus that Zika can also cause Guillain-Barre, a rare neurological syndrome that causes temporary paralysis in adults.
The connection between Zika and microcephaly first came to light last autumn in Brazil, which has confirmed more than 1400 cases of microcephaly that it considers to be related to Zika infections in the mothers.
Putting the Olympics risk in perspective, Dr Frieden said travel to the Summer Games would represent less than one quarter of 1 per cent of all travel to Zika-affected areas.
The CDC director called on Congress to deliver funding needed to fight Zika globally and to protect pregnant women in the United States and its territories, such as Puerto Rico, where officials expect hundreds of thousands of Zika cases.
Frieden thanked German drugmaker Bayer AG for promising a "substantial" donation to help fight Zika in Puerto Rico. The virus is spread by mosquitoes and through unprotected sex with an infected man.
With local US mosquito season about to begin, Dr Frieden said there was a narrow window of opportunity to mount an effective Zika prevention battle.
"That window is closing," he said.
The US Senate has voted to allocate US$1.1 billion of the US$1.9 billion the Obama Administration requested in emergency Zika funding, while the House of Representatives has promised only about US$622 million, much of that coming from resources earmarked for other health crises.
Dr Frieden said he hopes Congress "will do the right thing" and provide adequate Zika funding, as well as pay back what the agency already borrowed from other sources, such as money set aside to fight Ebola in 2017 and 2018.
"We need it back to keep Ebola from roaring back," he said.
There are already 279 pregnant women in the United States and its territories who have tested positive for Zika, health officials said last week.
"We need a robust response to protect American women and reduce the number of families affected [by Zika]," Dr Frieden said.
"Anything we don't do now, we will regret later."
The idea isn't gaining any traction here.
Olympic Committee chief executive Kereyn Smith said while Olympics decision-makers were taking the risk seriously, cancelling the event was not necessary.
She said there was a risk for any traveller going to international events, given the nature of the virus, so it was about taking precautions.
Ms Smith said Kiwi athletes on the whole had not raised many concerns about attending the games.
