But she was pivotal in driving Stuart and elder brother Alistair to cricket and golf.
"I'm not sure about that," said Stuart Duff with a laugh yesterday when asked if he agreed with his father's assertion.
In a 1995-96 cricket testimonial, the former Central Districts cricketer, who played for New Zealand A and, many felt, should have gone on to represent the Black Caps, paid tribute to his parents.
"The amount they have done for me is phenomenal," Stuart wrote. "Their encouragement, their financial support, their pleasure in success and understanding in failure have been endless."
As someone who followed his parents' career path, the Lindisfarne College schoolteacher considered himself blessed to have parental support which he is mindful of in his classrooms.
"Dad was a good cricketer himself and while he always played cricket with me when I was younger, I never felt as though I was being coached. It happened without my realising it.
"Mum passed on to me the value of persistence and her love and support have been a constant," he wrote in the testimonial, revealing brother Alistair didn't play cricket after first XI at high school probably because he had burned himself out in the never-ending backyard test matches.
When the boys were in their early teens, Daphne often drove him to Hastings club where junior golf co-ordinator, the late Murray Sparling, took them under his wings.
"He was very good with them. He walked around with them and gave them encouragement rather than tutoring," he said of Sparling.
For the record, Bill Duff, who played cricket with both his sons and still does 18 holes with Stuart at Bridge Pa every so often, attributes his golfing prowess to the three new clubs he has - a driver Stuart bought him, a three wood and a five wood - to replace his 25-year-old ones.
But everyone knows you still have to swing the clubs and fellow Karamu Golf Club members will attest to his ability to doing that as 14 handicapper over nine holes.
"In 1993 I won the junior men's club championship out there [Hastings club] so that was the highlight of my golfing career," Duff said with a laugh.