He has been there for the past five weeks, having built the digs himself, and has plans to renovate and extend.
"I've got quite a few more plans for it. I've got bags full of empty bottles and decided this morning when I got up that I'm going to plaster them into the walls and make a bottle wall.
"I'll do kind of a shop-like thing where you can sell coffees and smoked fish. Whatever we catch in the ocean we smoke on this fire."
Whakatāne District Council is aware of the situation and says it is working to resolve it.
Atkinson has also had a visit from Whakatāne Police.
"The police have been down to see me," he said.
"They said someone rung up the council and complained that I'd blocked off the peninsula to fishermen. I told them, well, I'm actually on an island. The land actually ends at that red dot," he said, pointing to the signal light near the Coastguard Whakatāne building.
He said there was once a channel for dinghies between where he set up his digs and the Coastguard building.
"Then some idiot decided to block it off so they couldn't get the small boats in when it gets rough."
Yesterday, several fishermen walked past the hut to fish off the rock wall at the river mouth.
The hut has been described on Facebook as an eyesore, along with speculation as to who might be living there.
Atkinson, 67, has called Whakatāne home for about the past year, though he has lived here before, around 20 years ago.
Born in Invercargill, he was raised on the Hokitika River. He said he was of Chinese, Irish and Cherokee ancestry, the latter being where the name 'Crazzy Horse' came from. He said he spent much of his time until the age of 14 panning for gold. Since then, he has become something of a jack of all trades.
"I've spent most of my life in the bush, either that or a truck. I drove road trains in Australia for 17 years."
He said he had also been a shearing contractor in Gisborne for 24 years. "I'm used to living rough. I've lived like this all my life. I've lived in some pretty rugged shearing quarters."
He also said he had his off-shore masters ticket for commercial fishing boats, which was his most recent line of work. He described himself as a bushman and said the late author Barry Crump was a good mate of his. "I knew Barry really well," he said.
He added he wanted to thank the many locals and fishermen who had made him welcome, helping him out with items to make his hut more comfortable.