Organisers said forecast conditions had become "significantly worse in the past 12 hours, with information coming to light that has not been available to us before the event kicked off" yesterday.
"After consultation with local and regional fire authorities and other emergency stakeholders, it is clear that we have no other option," organisers said on Facebook.
"While conditions are fine at time of writing, we are taking the opportunity to move everyone offsite safely and in good time."
The festival kicked off yesterday with a highlight a performance from Aussie music royalty John Farnham.
The legendary singer's son, Jim Farnham, posted a video of the Falls crowd in full voice singing along to his signature tune You're the Voice.
More than 9000 people are currently camping at the festival grounds in Lorne, a small surfing town on the Great Ocean Rd southwest of Melbourne.
"For those 9000 currently staying onsite we ask you calmly pack up your campsite and return to your place of origin," organisers said on Facebook.
"There is no need to rush, please ensure you have a sober driver."
Festival-goers have taken to social media to vent their disappointment.
One punter, Chloe Heath, can be heard in an Instagram post saying "It ended before it began" as she films the long queue of cars trying to leave the campground.
The cancellation means punters will miss out on performances from headliners Halsey, Vampire Weekend and Disclosure.
Halsey, a US singer-songwriter, said she was "super devastated" by the cancellation.
"It sucks, but I'm also happy because it's important that Falls is keeping you guys safe in regards to extreme weather and the potential fires," she told her fans on Instagram.
"But with that being said, I know how disappointed some of you are and you guys mean the work to me so I'm going to do everything in my power to find some way to make it up to you.
"Your safety is my number one importance, but I also really hate disappointing you."
Drivers are asked to avoid Lorne or other coastal townships and instead leave via Mt Sabine and Deans Marsh roads onto the Princes Highway.
"Do not look to camp in the Otways or any forested area, as the fire risk is extreme and there are forecast wind gusts of 90-100km per hour with possible lightning," organisers said.
All ticket holders will receive a full refund including booking and payment processing fees.
Sister Falls Festival events in Byron Bay, NSW, Marion Bay, Tasmania, and Fremantle, Western Australia, will go ahead as scheduled.
"We are gutted to make this call but the safety of our patrons, artists and staff is our main priority," Jessica Ducrou, chief executive officer of festival organiser Secret Sounds, said, adding thanks to Surf Coast Council, the State Emergency Service, Victoria Police and the Country Fire Authority.
"We would also like to thank our Falls family who work year round and over the last few weeks who have put their heart and soul into Falls Lorne, we are forever grateful," Ms Ducrou said.
"Please take care getting home safely."
It is with a heavy heart we have had to cancel the remaining days of The Falls Festival in Lorne due to the predicted...
Posted by Falls Music and Arts Festival on Saturday, 28 December 2019
