"I thought the second one [knee injury] was going to break him, I've got to be honest," explains Storm football director Frank Ponissi.
"I was with him at the AAMI Park medical centre and it was incredibly hard. I can't remember the actual words he said but it was along the lines of 'I just don't know if I can do this again'.
"Quite understandably, he was questioning himself and where he was going."
That self-doubt and mental anguish is apparent within a post on the 25-year-old's personal website, in which he details his struggles throughout two-and-a-half years of relentless rehabilitation, while preparing for yet another comeback before this season.
"I've had to put my hand up and own that I never really deserved the privilege to play for Melbourne Storm," he wrote.
"I'm coming back from injury this time with my youthful arrogance smashed to pieces but I'm a better person for it.
"And I can say now, to the entire Melbourne Storm family, with complete sincerity: I think I'm ready to play for you. I think I'm ready to earn your respect."
The road back involved approaching the club seeking a new contract. A minimal deal, made up mostly of match payments, was as good as it got.
He had to fight most of the season before he was finally rewarded with a first grade spot, 847 days after his last appearance.
"[Storm coach] Craig [Bellamy] was never going to give him a jersey and Duff knew that he doesn't just hand those purple jumpers out for emotion," Ponissi explains.
"He had to earn it but it took him until round 20, back in New Zealand - it's interesting isn't it - against St George in Napier. It was a special moment.
"There was a lot of emotion going into that game with Tohu Harris' connection to Napier but also the Matt Duffie comeback, and that night was one of the better performances and wins of the year.
"For a bloke that's had a lot of success - played for his country and played at a high level - he's just had a lot of setbacks but he keeps bouncing back."
While his body required a great deal of mending, Duffie insists his comeback wouldn't have been possible had he not sought professional help to overcome his mental demons.
"I probably wouldn't be sitting here unless I had gone out and sought that help, that was a big thing for me," Duffie said.
"Getting the head right is a massive, massive part of it.
"I've never claimed I've done this whole thing by myself and I never will. There has been a whole heap of people who have helped me along the way. It would have been that step too far if I didn't have that support around me."
Unsurprisingly, the entire experience has had a lasting effect on the former St Kentigern College First XV and athletics star.
New Zealand rugby may have rued the day they let him get away, but the Blues can be sure they are getting back a unique player with the hardened spirit and attitude of a true professional.
"He's been mentally tested over the last few years with his injuries and I think his whole philosophy on life, I wouldn't say changed, but he's become more of a deep thinker," said Ponissi.
"He's always been a young man far more mature than his age. When he walked in here, he wasn't your typical 18-year-old. He was still very popular among his peers but he had that ability to think outside the square as a youngster in his early years of NRL.
"But the last few years, with what he's gone through, he's spent a lot of time on his own at various stages. I think it's given him a checklist, not just in his career, but also in his life."
The decision to leave the Storm was a difficult one, but a desire to return to rugby and the challenge of playing fullback and learning under Blues coach Tana Umaga proved enticing.
"I have always held fond memories of my days playing union and more recently it has occurred to me that in some respect my body is more suited to union," Duffie said.
"Tana has both the depth of talent coupled with broad experience to be a positive influence on those he coaches. I hope to learn much from him. Plus the possibility of being considered for fullback duties leaves me excited."
With the wisdom of a player who's traversed the highs and lows of a rollercoaster career, he is guarded when asked about his All Blacks aspirations. Behind his words, however, lies a quiet determination and knowledge that the best times will come if they are earned.
"There would be few players who haven't dreamt of playing for the All Blacks but I'm smart enough to appreciate that these things are best evidenced by performance rather than loud talk. So you won't find me talking up possibilities long before I've earnt them."
