Junior later said that he felt abandoned and was bullied at school, and suggested a trust fund should have been set up for him.
But the bitterness appeared to have softened somewhat around the time of Ali's death, as he joined his sisters, half-sisters, and even his adopted brother Asaad Amin Ali, 30, at the hospital.
Ali and Lonnie adopted Asaad as a baby and he was very close to the former heavyweight champion. Following his adopted father's death Asaad, a baseball coach, said: "I'm forever grateful for everything you ever gave me."
Also at the hospital was the champion's brother Rahaman Ali, 72, who had previously publicly claimed in a US magazine that Lonnie had cast him out, and worse, although he later said he was misquoted.
After seeing his older brother for the last time in the hospital Rahaman said some of Ali's final words were about "going to Allah".
Rahaman said: "He said to me, shaking, 'Rahaman, how do I look?' I said 'Even with all that shaking you look the same to me.'
"He said to me 'I'm in no pain. No pain. Don't cry for me Rahaman. l'm going to be with Allah. I made peace with God, I'm OK.'
Rahaman added: "He was a wonderful angel. I miss him but I will see him again in heaven. He'll always be with me in my heart, in my bones, in my mind.
"My black brother who lived in Kentucky became the most famous man in the world. That's a blessing from God, and I was there from the beginning."
Bob Gunnell, a spokesman for the family, said relatives had shown "dignity" in putting aside any squabbles to be by the bedside at the Osborn Medical Center in Scottsdale, Arizona.
He said: "It was a beautiful thing to watch which displayed all that was good about Muhammad Ali. The champ would have been very proud of his family."
Ali had been taken into hospital on Monday night after which his condition deteriorated. The official cause of death was septic shock due to unspecified natural causes.
He lived nearby with Lonnie in a relatively modest $2.1 million six bedroom bungalow on a gated estate made up of 72 homes amid cacti and palm trees, looking on to a golf course.
Other members of the family have defended Lonnie who married Ali in 1986 and, along with her sister, cared for him throughout his battled with Parkinson's disease.
Their mothers Marguerite Williams and Odessa Clay were best friends and Lonnie first met Ali when he was 21, and she was aged six.
Speaking in 2010 she said: "I think I've always loved him. I knew, at the end, I was going to be the one married to him. It was like a road map."
She said Ali's children "do not see their father as much as they would like" but it was difficult because "with this disease you never know how long he can communicate".
Mrs Ali gained a masters business degree from UCLA and has been credited with turning around the boxer's chaotic finances.
She said she was "somewhat stunned" by his financial condition after he retired.
"I thought he should have been better off given the nature of who he was and the people he was taking care of," she said
"It was understandable. Muhammad is not a businessperson. Fortunately, as he got older, a little bit smarter and wiser, he has been more prudent and frugal."
Ali's estate is unlikely to benefit form the kind of money-spinning merchandising legacies of other celebrities like Michael Jackson or Prince.
In 2006 he sold 80 per cent of the rights to his image for $50 million, including the rights to trademarks like "Float Like A Butterfly, Sting Like A Bee," "Thrilla In Manila," and "Greatest Of All Time.".
The rights are currently owned by the same company that licenses the images of Elvis Presley and Marilyn Monroe.
However, the family still has input into how his image is used and has been very selective about licensing deals.
Among only a few dozen companies allowed to use Ali's image are Adidas, and an electronics company for video games in which players can box the champion.
