"We have looked at the logistics and construction period and I have made this decision because I was assured that we can definitely complete construction on time. We must control costs as far as possible."
World Rugby said Japan's decision had caught them out as they had been assured a number of times by the Rugby 2019 organising committee and Japan Sports Council there was not a problem.
Japan's bid for the 2019 RWC was awarded in 2009 giving them what should have been ample time to sort out the infrastructure.
Now plans for the new national stadium have been shelved, officials say they will not be able to complete a replacement until 2020.
Sports Minister Hakubun Shimomura said a new bidding process would aim to have a design sorted in six months with the construction due to be completed in 2020.
IOC president John Coates said the national stadium review would not affect construction for the Olympic and Paralympic Games and the IOC would work with the 2020 Tokyo committee to ensure there were no issues.