But he eventually got back into dressage and qualified for the Beijing Olympics.
He and Whisper made a stir and gained a following in both Japan and Germany, where he is based so he can train with his Dutch coach.
In Japan, fans call Hoketsu "the hope of old men".
But his journey has not been without sacrifice.
"I have not seen my wife, Motoko, for more than a year," Hoketsu told Yahoo Sports. "It is difficult to be away from home for this long as an old man and I owe everything to her patience and understanding."
At first it wasn't even clear if he intended to compete in this year's Games. In November, Hoketsu said he wasn't even considering going to London, because Whisper was suffering from severe tendinitis.
He said he just wanted his horse to heal so they could retire in Japan together.
But not not only did Whisper recover, but then they both went on to qualify for the Games in March.
Although Hoketsu gets up early every morning, he admits he doesn't stick to the dietary regime you might expect from an Olympic athlete.
"I eat what I want to eat and drink as much as I want to drink," he said.
"People might expect that I am able to participate for so long because I have special habits. But my secret is to have a good life, enjoy yourself and do the things that make you happy."
Hoketsu is not the oldest Olympian in history - Sweden's Oscar Swahn won his sixth medal, a silver, in shooting at the 1920 Antwerp Games at the age of 72.