The limited edition bars only appeared on shelves a few weeks ago and have been selling out.
Cadbury said "small flexible pieces of food grade plastic" had been found in the product due to a "machinery fault".
Ingestion of the plastic could cause a "minor injury," the statement said.
Only products with a best before date of 17 and 21 January 2019 are affected. Other Caramilk bars and other Cadbury blocks are safe to consume.
The recall couldn't have happened to a worse product given Caramilk, which is made from caramelised white chocolate and tastes like sweet condensed milk, is in the midst of a cocoa craze.
The New Zealand produced bars have been snapped in droves and have been turning up on eBay. On the auction site, a pack of seven Caramilk bars is being touted off for $150 — that's more than four times the recommended retail price. And it's not the only eBay listing for the bars.
"Forget Bitcoin, Caramilk is proving to be a wise investment," said Sara Czarnuch on social media.
Caramilk, which was once a regular in the Cadbury line-up, first reappeared in New Zealand last year and sold out in three weeks.
Savvy Kiwis were said to be selling the bars online for huge mark -ups to deprived Aussies, and some went for up to $50 a pop, reported the New Zealand Herald.
But are they really all that popular, or is it all a gimmick with few bars on sale?
Cadbury spokesman Paul Chatfield insisted to news.com.au the rush was all due to customer demand.
"When Cadbury Caramilk was brought back for a limited time in New Zealand last year, we saw Kiwis selling blocks online to Australian buyers for over $30 a block.
"Given the demand from Aussie consumers, we've now brought a limited quantity to Australian retailers and feedback from the first stores to have the block on shelf is that they're disappearing fast."
However, there are off the shelves now for all the wrong reasons.
Customers don't need a receipt to return the bars.