"They almost ended the relationship, but Brad stayed," the source added.
The report suggests the 54-year-old actor is "not on good terms" with sons Maddox and Pax following his split from the actress and "has accepted that he'll likely never be able to repair their relationship".
"[Maddox has] always been headstrong and protective of his mother," the source said, adding Maddox was Jolie's "closest confidant" during the fallout.
Shiloh has "seemed the most eager and willing [of the four eldest] to work on things with Brad," the insider claimed.
Zahara now has an "improved" relationship with her dad, they added.
Twins Knox and Vivienne meanwhile "have been sheltered from a lot of the issues", the source said.
Jolie filed for divorce in September 2016 after an alleged altercation involving Maddox and Pitt on a flight from France to LA.
Late last month, lawyers revealed the star couple had finally come to a custody agreement regarding their six children Maddox, 17, Pax, 15, Zahara, 13, Shiloh, 12, and 10-year-old twins Vivienne and Knox.
According to Us Weekly, both actor's parenting skills came under fire following assessments from social workers.
The source said that while Brad made huge strides "addressing his anger and psychological issues", one social worker "felt he had trouble accepting personal responsibility and often blames others for his actions."
Angelina meanwhile was allegedly chastised "for indulging the kids all the time and creating an unstable living environment."
One case worker reportedly believed she was currying favour with her children, and that it hurt Brad's relationship with his kids; "But the worker didn't think Angie was trying to actively sabotage the relationship."
The insider also claimed that Brad and Angelina will never see eye to eye.
"Angelina hasn't offered any concessions about her actions that have contributed to the situation," the insider added.
"Angelina has a lot of emotional baggage from her own unstable childhood," the source concluded, describing her as "extremely complex."
Pitt and Jolie married in August 2014 but split in September 2016.