It is the first time New Zealand have lost to Japan in any form of the game.
The eventing team were in the gold medal position with one ride left in the show jumping.
Sir Mark Todd dropped four rails and the team slid out of medal contention by 3.5 penalty points - or less than one rail.
"It's disappointing for me and the whole team," Todd lamented. "I feel more disappointed for the team who had put us in a position to win a medal and it was up to me to deliver.
"My horse is normally a good jumper but he went out and froze in the atmosphere. There was nothing much I could do, I tried to relax him and stay calm, but he kept putting his head up and hollowing his back. It was awful."
New Zealand rowers had mixed results.
Both double sculls crews missed out on their finals, including world champions Zoe Stevenson and Eve MacFarlane.
However, the men's pair of Hamish Bond and Eric Murray as well as the men's lightweight four of James Hunter, Alistair Bond, Peter Taylor and James Lassche qualified for their respective finals.
Single scullers Mahe Drysdale and Emma Twigg both reached their semifinals.
The men's Black Sticks added to the disappointing results. They lost 3-2 to world number 11 Spain - the second loss of their campaign - and now need wins in their final two pool fixtures against Brazil and Belgium to remain near medal contention.
New Zealand's struggles in the pool continued in the morning session with three swimmers failing to advance out of the heats.
Matt Stanley finished last in his 100m freestyle heat in a time of 50.14s. Helena Gasson was second in her 200m butterfly heat but was 25th overall. Glenn Snyders won his 200m breaststroke heat in 2:12.47s which ranked him 23rd, missing out on the semifinals.
On Guanabara Bay, Finn sailor Josh Junior began with 18th place and did not finish his second race.
Sam Meech lies third in the Laser, one point adrift of second on 14. He recorded a fifth and sixth today but has discarded the 19 points he collected in race one.
Still to come today/in progress:
Rugby sevens: Pool game New Zealand men v Kenya, 8.30am
Football: Women's group game - Football Ferns v France, 10am
Cycling: Women's road time trial, Linda Villumsen, 11.30pm
Rowing: Women's lightweight double scull semifinals (if qualified) - Julia Edward and Sophie MacKenzie, 11.50pm.
