The Tokyo Games are set to open on July 23.
Fans from abroad were banned several months ago. Officials say local fans will be under strict rules. They will not be allowed to cheer, must wear masks, and are being told to go straight home afterwards.
Organisers say between 3.6 and 3.7 million tickets are in the hands of Japanese residents.
Having fans in the venues presents a risk of spreading the Covid-19 infections, and not just at the venues, since it causes more circulation on commuter trains, in restaurants and other public spaces.
Tokyo and other areas are under "quasi-emergency" status until July 11. This replaced a tougher full state of emergency that was in effect until last weekend. The new rules will allow restaurants to serve alcohol during limited hours.
Prime Minister Yoshihide Suga, who has favoured allowing fans, said before the official announcement that he would bar fans if conditions change.
"If a state of emergency is necessary, I will be flexible and open to no fans in order to achieve that the Games give top priority to safety and security for the people," Suga said.
"In case of a state of emergency, it is quite possible ... for safe and secure (Games) I will not hesitate to have no fans."
He said he took "seriously" Omi's recommendations but did not follow them.
Tokyo Governor Yuriko Koike also said before the talks that fans may need to be banned if conditions changes.
"Under this Covid-19 pandemic, people in Tokyo, people in Japan, are feeling very uneasy. We have concerns and the experts are also making recommendations about the risk of spreading the infection," Koike said.
"If there should be a major change in the sanitary situation, or infection situation, we need to revisit this matter among ourselves and we may need to consider the option of having no spectators in the venues."
In recent polls, support seems to be increasing for holding the Olympics, though opposition is strong, depending how the question is worded.
An Asahi newspaper poll of June 19-20 of almost 1500 people showed 62 per cent supported another postponement or cancellation of the games. But about one-third supported holding the Olympics, up from 14 per cent in May in the same poll.
In the same survey, 83 per cent said they "feel uneasy" that the Olympics might spread the virus. The poll said 53 per cent wanted no fans and 42 per cent said attendance should be limited.
The seven-day average for new infections in Tokyo is about 400 daily. The curve has flattened but health officials fear the Olympics and new variants will drive it up.
About 6.5 per cent of Japanese are fully vaccinated, and 16.5 per cent have had at least one shot, according to figures from the Prime Minister's office. More than 14,000 deaths in Japan have been attributed to Covid-19.
