"Back then it was a hard decision to make but I think it was the right one because my mind wasn't with it.
"I've had a bit of a yo-yo experience with my weight and that was another factor why I ducked out of the sport because I just couldn't manage my weight properly."
But a year ago, Bryant decided he wasn't done.
"I didn't want to go out thinking I had a shit boxing career because boxing's done a lot for me," he said. "So I had a think, if I'm going to do it this time, I'm going to do everything as best as I can do it. So I haven't had a beer in basically a year, I've stopped playing rugby and just concentrated on boxing and my health. I've lost 20 kilos."
From his days as a front-row forward where he weighed around 110kg, the 1.81m Bryant trimmed down to a lean 90kg and has fashioned a 3-0 record as a professional cruiserweight boxer this year with two of his wins coming via knockout.
The 33-year-old has done enough to earn a spot in November's Super 8 tournament on the North Shore where eight cruiserweights will be paired off to meet in four quarter-finals over three three-minute rounds with a winner to be found on the night.
The card will be headlined by Kali Meehan and Shane Cameron who will square off for the vacant WBA Oceania heavyweight title. Rising Kiwi cruiserweight David Aloua will also meet Australian Daniel Baff in a bout where Aloua's regional titles will be on the line.
The Super 8 tournament promises to be competitive with Australians Daniel Ammann, Anthony McCracken and Pitt confirmed for the bill. Kiwi Monty Filimaea will also be involved, with three more to be added.
Bryant, now a father of four, hasn't allowed himself to think too far ahead about what a potential victory could do for him, although he acknowledged it could open doors to some money-spinning fights across the Tasman.
"You don't want to think too big, I'm just thinking about training, working and family."
Bryant will continue his work as a shepherd in Pahiatua alongside his twice-daily training regime as he leads in to the Super 8. He will also link with Cameron for a few sessions as the pair are both coached by Henry Schuster.