Plumes of smoke billowed from the one-storey, five-bedroom house as several inmates looked on from housing units at the prison across the road.
Their cat George died after inhaling a lot of smoke but Horsfall, a top sheep dog trialler who won a North Island Sheep Dog Championships at Te Aute last month, said his trial dogs were fine.
"My wife lost a lot of her tapestries she'd done since she was 16 and I lost all my trophies from dog trials but probably more importantly nobody was hurt except for our cat."
Speaking from the charred remains of his family home yesterday, he said the lounge, dining room and kitchen were "absolutely stuffed" and most of the bedrooms were left smoke damaged and wet.
"It's a horrible thing to happen, I wouldn't wish it on anybody to be quite honest. It's not a good feeling. It goes from your home to just a house."
Originally from Whanganui, Horsfall and his wife moved to Hawke's Bay and bought the house and accompanying chicken farm nearly 20 years ago.
He said dog trial trophies, tapestries and other precious possessions they had accumulated over their lifetimes were completely destroyed in the fire.
"It's like the loss of part of you because it's been our life for a long time, stuff since we first got married and kids stuff, books and photographs you can't replace.
"In a lot of ways it's a bit like somebody died but you get over it eventually, rebuild and move on."
Fire Service specialist fire investigator Mike Finucane said the fire started in the lounge before spreading to the rest of the house.
While the fire remained under investigation a definitive cause had not been determined and it was likely to remain that way, he said.
Horsfall said he was fully insured and the support they had received from family, friends and emergency services was incredible.
"The police and the fire brigade were marvellous, they were unreal. I can't speak highly of the job they did enough."