The claims come as Pitt has reportedly warned Jolie he is ready to fight for joint custody of their children amid reports she told him "I can't do this any more" before filing for divorce.
Sources close to Pitt told TMZ that he is angry at his estranged wife for allegedly spinning stories about him being a potential risk to their kids because of "anger issues and substance abuse".
Pitt reportedly said the actress has "unleashed hell" because the entire family has become targets for the paparazzi while reports have also claimed that a shocking "incident" may have prompted the divorce.
Pitt said he wanted to keep the divorce under wraps for the benefit of their children.
Over the last few days, Pitt has had conversations with Jolie pleading for her to handle the divorce "like adults", sources told the site.
The Hollywood stars' home is now surrounded by photographers, with the Los Angeles Police Department patrolling regularly because of the media frenzy.
Jolie filed for divorce from Pitt on Monday for "the health of her family", two years after the couple wed at their French estate Chateau Miraval.
The actress filed papers citing irreconcilable differences as the reason for the split and asked for physical custody of the couple's six children - Maddox, age 15; Pax, aged 12; Zahara, aged 11; Shiloh, aged 10; and twins Vivienne and Knox, aged eight.
A source close to the couple told TMZ that Jolie, 41, made the decision to file because of the way "Brad was parenting the children," adding that "she was extremely upset with his methods".
That same source claims Jolie became "fed up" with Pitt's consumption of "weed and possibly alcohol" mixed with his "anger problem".
It has also been revealed that Pitt was blindsided and was told that Jolie was divorcing him just a day before she filed the legal papers, a source told the Mirror.
Pitt allegedly visited his lawyer to discuss how to move forward in Los Angeles hours after Jolie met with her lawyer.
The source said the pair had been planning a trip to London but it was canceled Monday, the same day Jolie filed for divorce.