They say they are "pretty frustrated" as the structure takes up nearly half their outdoor area.
Chapel owner Christine Merritt says she has been trying to get the building listed as a heritage site.
"I would like it repaired because it's an icon," she told Fairfax.
"I've been trying to do something about it for the last 12 months. It's not like I've been sitting on my arse," she added.
Merritt erected a safety fence around the timber propping near the front and side of the building, which seems to have created further issues for the Graham family as the fence means the council no longer considers the chapel "a dangerous building" and, as such, won't act on it.
The fence does not include the Graham's side.
"How's that going to protect me and my family from the building falling down on us? She doesn't want to know about it. It's not her concern," Graham said.
Head of regulatory compliance, Tracey Weston, says the chapel owner is within her rights and has followed safety protocol.
She says the dispute regarding boundary encroachment isn't a council matter - neighbours have to sort it out between themselves.
The war is far from over as Merritt says she timber propping structure will stay there for as long as it takes her to find funds as well as an engineer willing to take on the job.
"Stuff them," she reportedly said of the neighbouring family.
