This was backed up today, with the ABC reporting that the Australian heads of the immigration and defence departments told the Turnbull Government earlier this year not to join BRI.
However, the Department of Foreign Affairs and Trade were reportedly broadly in favour of joining.
Beijing's massive plans, which were first unveiled in 2013, involve the reviving of an ancient land and ocean silk trade routes.
It has already spent billions of dollars on new infrastructure projects for roads, railways, ports and maritime corridors.
Clarke said Australian politicians are torn between our commitments to the US and the growing Asian superpower - but they must make their minds up sooner or later.
"The key issue is whether Australia can have its cake and eat it too when it comes to China," he said.
"They want to maintain a strong economic relationship with Beijing, but at the same time maintaining a very strong commitment to the US and the regional order it has set out since the Second World War.
"BRI brings that dilemma out even more than in recent years - particularly with the election of Trump, the inward turn of the US and its questioning of commitments throughout Asia.
"That is something we should take very seriously. It's very clear that there are challenges and opportunities for Australia stemming from BRI."
China has already pledged more than A$100 billion (NZ$112b) and it will be completed in a series of bilateral projects - such as the A$46b China-Pakistan economic corridor, or a A$5.2b railway that will eventually link Laos, Thailand, Malaysia and Singapore.
Combined, the plan will include more than 60 countries and 4.4 billion people, passing through Asia, Europe, Africa and the Middle East, covering up to 40 per cent of global GDP.
Dr Clarke the project's impact on Australia is unclear at the moment - but there will be both challenges and opportunities.
"We can't simply rely on either doing nothing or doing what we've done in the past which is always to sort of hedge our bets and rely on the US for security and China for economic growth - it misses the point," he said.
"The point is BRI is reshaping the geopolitics of the region, so we need to come up with some coherent response to it."
The Chinese Embassy said the BRI's "overall vision" was to build a "road of peace, prosperity, opening up, innovation and civilisation" that is open for collaboration with other countries. It's inspired by the ancient Silk Road, it said and over the next five years, China would spend up to A$1 trillion in participating countries.
Mr Clarke said BRI is a "fairly crude" geopolitical strategy which is also driven by China's own economic concerns.
"It's also a bit of rhetorical window dressing and a soft power strategy designed to portray China's rise in a positive light," he said.
The enormous venture has been called the "biggest story" in Asian business that could redefine global trade. Combined with US President Trump's decision to pull out of the trans-Pacific Partnership, some suggest it could signal the start of China becoming the leader on globalised exports.
Last week, the superpower added more flesh to the bone of its grand vision - as Xi outlined plans to become the world's biggest superpower within the next 30 years.
Opening a five-yearly national congress, the leader set out his time frame for the country to become a "global leader" with international influence.
In his epic 3 ½ hour speech, Xi urged a reinvigorated Communist Party to take a stronger role in society and economic development to better address the nation's "grim" challenges.
Hailing the start of a "new era", Xi outlined a vision in which the party would lead China on the road to becoming a "great modern socialist country" by the mid-century.
"The Chinese nation ... has stood up, grown rich, and become strong - and it now embraces the brilliant prospects of rejuvenation ... It will be an era that sees China moving closer to centre stage and making greater contributions to mankind," he told the national congress.
