The accident happened at lunchtime, so not many workers were on site when the crane collapsed on top of the metal structure, causing the deadly domino effect.
It wasn't the first problem with World Cup stadiums in Brazil.
One worker died during construction of a stadium in the capital Brasilia last year and another in the Manaus venue in March. Also in March, heavy rains flooded the construction site of the Maracana Stadium, forcing the cancellation of a FIFA inspection visit at the time. In May, a small part of the roof at the Salvador stadium fell in after it wasn't able to sustain the large amount of water that settled on top of it.
The Sao Paulo stadium, which cost nearly $US360 million, will seat nearly 70,000 people for the opener on June 12, when Brazil will play. About 20,000 seats will be temporary and installed only for the World Cup.
The venue is scheduled to host other five matches, including a semifinal. It was initially expected to be built for the Confederations Cup this year, but delays with financing for the venue prompted authorities to scrape the stadium from the World Cup warm-up tournament.
Brazilian club Corinthians, which owns the stadium, released a brief statement "lamenting" the accident.
Constructor Odebrecht said it was already investigating what happened.
Brazil is running against time to deliver the last six World Cup stadiums by the end of the year, although work at the Itaquerao was advanced compared to the other venues. FIFA has been pressuring local organizers to make sure all venues are ready by the December deadline so all test events can take place in time for the World Cup.
FIFA said it would not accept the same delays that plagued stadium construction before the Confederations Cup, when only two venues were delivered in time.
"FIFA and the LOC have learnt of the death of workers at the Corinthian's Arena site in Sao Paulo with great sadness," FIFA said in a statement. "We wish to send our heartfelt condolences to the family of the workers who tragically died today."
Football's governing body said the "safety of workers is the top priority" to World Cup organizers.
"We know the safety of all workers has always been paramount for all the construction companies contracted to build the 12 FIFA World Cup stadiums," it said in a statement. "The local authorities will fully investigate the reasons behind such a tragic accident."
FIFA President Sepp Blatter said he was "deeply saddened by the tragic death of workers" at the Corinthians stadium. "Our heartfealt condolences are with the families," he said.
FIFA Secretary General Jerome Valcke said he was "extremely shocked by the news from Sao Paulo."
"Our thoughts are with the families of the victims of this accident," he said.
- AP