Much of it came from Jolie, who deliberately distanced herself from Voight in her adult years. Yet it wasn't always like that. Voight recalls one year at the Oscars - 1986 - when he was up for Best Actor for his ruthless bank-robber in Runaway Train and was expected to win. He went with Jolie and her older brother, James. "My name was not announced - [and] my son responded - he just crossed his arms and slunk down in the seat and was really upset that a great injustice had been served! And my daughter to my left put her hand on my arm and said, 'Don't worry Dad, you're wonderful!' It was so sweet."
Voight, all silver hair, moist blue eyes and a hulking 6ft 2in frame, seems touched by the memory.
"As adults", he and Jolie only made one movie together, 2001's video game adaptation Tomb Raider, although as he reminds me, when she was seven, she made a brief appearance in Hal Ashby's comedy Lookin' To Get Out, playing his daughter. "It was a nice little scene. I didn't know that she would go on to be this icon." Indeed, it would be impossible for any father to imagine his child becoming one of the most photographed and talked-about women in the world.
It was shortly after Tomb Raider where it all went wrong, when Voight publicly blamed his divorce for provoking Jolie's "serious mental problems", claiming "she was never normal". Then on her second marriage, to actor Billy Bob Thornton, Jolie responded by cutting ties from Voight, proclaiming, "It is not healthy for me to be around my father."
It was only when Bertrand died from ovarian cancer in 2007 that Jolie softened her stance, reportedly at the urging of her partner Brad Pitt, and brought Voight back into the fold.
Since then, he has been pictured with her at the LA premiere of her directorial debut In the Land of Blood and Honey and spent time in London with the Brangelina brood - his six grandchildren - when Pitt was filming zombie movie World War Z.
Angelina Jolie. Photo / AP
So how does he feel towards her now? "Well I'm always happy when she's happy," he says. "And when she does good. And there's a lot of good that she's done, y'know? She does things quietly to the side. I'm very proud of her. And she's a good director now too."
Much was made recently when Jolie accepted the Jean Hersholt Humanitarian Award in Hollywood for her charitable work, singling out her late mother in her acceptance speech and ignoring her father (who was in the audience).
But Voight, like Bertrand and their daughter, has quietly given back, making films on Chernobyl, the plight of the native Americans and the Rainbow Warrior. He organised marches for the homeless and supported Vietnam vets (echoed in his Oscar-winning role in Coming Home). "I've had a lot of causes I was concerned about," he says. "Always my mind is on some of the problems that I see in the world."
It was during the 1980s that Voight became increasingly concerned, only shooting four movies. "I was pretty picky," he concedes. No kidding: this was the man who'd turned down the leads in Love Story, Jaws and Superman.
Even when he was making Coming Home, he told director Hal Ashby, 'why don't you just get Al Pacino?"'
Perhaps it was paranoia: even at his height, Voight could never steal a march on Pacino and his peers. Like his father Elmer, who made his career as a golf caddie, he was always carrying the clubs for others.
Ironically, it was his mother Barbara who inspired Voight to get back into Hollywood. She died of cancer in late 1995, aged 85. But in her final years, Voight wanted to spend as much time as possible with her. It's why he took on Mission: Impossible, so he could take her to Prague. "The making of that movie was a real blessing for me. I didn't have too much time filming, so I was able to focus on her. She didn't sleep much - four hours a night - and she'd be up wanting to do something. After that, she said 'Well, what's next?' So I had to go out and look for more films. That's how I got back into it."
Rediscovered by the likes of Michael Mann (who cast him in Heat and Ali), Oliver Stone (U Turn) and Francis Coppola (The Rainmaker), Voight finally became the character actor he'd always resisted becoming.
Born in Yonkers, New York, his early years had indeed been as a jobbing actor - in television shows like Gunsmoke and Naked City - before he landed the self-styled gigolo in Midnight Cowboy.
"I trace my lineage back to the character actors, not to the great stars. And there's a difference."
It's why Ray Donovan brings him full circle, five decades after he started off in television. He clearly revels playing the ruthless Mickey, who opens the very first episode by leaving prison and killing a priest. "When I do it right, when I catch it, it feels great," he says, his lips smacking.
He's also got a film lined up - to be directed by Andy Garcia - about writer Ernest Hemingway and his relationship with the boat captain who inspired The Old Man and the Sea . Taking over from Anthony Hopkins, who dropped out, it's yet another real-life figure to add to a collection that includes the title role in TV movie Pope John Paul II and President Roosevelt in Pearl Harbor.
"It's a very poetic piece," he says. "I'm discovering so many things about him and reviewing his work as an older fella, and it's very rich." Somehow, he seems perfect casting.
Who: Jon Voight, veteran actor and father of Angelina Jolie
What: Ray Donovan series 2
Where: SoHo, 8.30pm Wednesdays
- TimeOut / The Independent
