The toxic deposits are deemed to pose a low risk to human health.
A soil contamination report on the Cadbury site, commissioned by former owner Mondelez before it sold the property to the Ministry of Health, said the land could either be left untreated under the existing concrete slab, or removed.
Demolition of the factory is now about halfway complete, and the adjacent Wilson block is now mostly cleared.
New Dunedin Hospital programme manager Mike Barns said no decision had been made about what to do with the soil.
Leaving the site capped would speed up the project time line, and clearing it would reduce the amount of material being recycled by the demolition crew as contaminated material would have to be correctly disposed of, he said.
"There are a lot of benefits in not puncturing it and removing it," Barns said.
"There are some options to only puncture it with our foundations and to keep the slab there. That's certainly something we are looking at.
"Total removal, it's an option but it doesn't provide us with as much benefit as the other options provide us with."
A high level of lead is in the soil at a historical foundry and current workshop site, and arsenic was found where a historical sawmill stood and a range of industrial processes took place.
"The concentration of contaminants do not, however, suggest a site-wide level of significant risk to human health or to the environment," the independent report said.