But there was also much more.
It is easy to be seduced by Norfolk's many and simple charms: the island invites you to poke around its small side roads where you find the tasting room of Norfolk Island liqueurs and Cascade Soft Drinks, which makes Plum Cola and wonderful Creaming Soda; or discover a house made out of bottles.
You can shop duty-free: for shoes you go to a place called Imelda's of course; the marvellously named House of Scruples is worth a visit; you can sit and drink locally grown coffee at a cafe in town; have locally made beer at The Brewery; or snorkel at beautiful Emily Bay.
You can also happily do not much at all. But Norfolk has such a compelling history you are inevitably drawn to the Bounty Museum, the cemetery which is beside one of the world's best located nine-hole golf courses, or the impressive Cyclorama which tells the story in the round of the Bounty and how the descendants of the mutineers came to this fascinating, if remote, island.
And there is more.
In her remarkable book, The Emperor's Last Island, about Napoleon's final years on St Helena, the British author Julia Blackburn writes: "A place can be haunted by the people who knew it long ago and who stared at the stones under their feet, the leaves on the trees, and out at far distances and horizons that encircled them.
Blackburn may well have been writing about Norfolk Island, a lonely but lush rock in the western Pacific which is only 8 by 5km in dimension, and about 1000km from Auckland.
Norfolk Island was discovered in October 1774 by Captain James Cook, who wrote kindly of it in his journal: plentiful bird and fish life, good soundings, fresh water, and the native pines he thought useful for masts - although they subsequently proved not to be.
Cook didn't stay and he perhaps caught Norfolk - which locals pronounce Nor-folk - on a good day. The French captain La Perouse 11 years later couldn't land because of heavy swells and described the island as only a place for angels and eagles to reside in. Even these days, the locals tell me, their fishing boats can get out only about 60 days a year.
But it was what happened shortly after La Perouse gave up on the place that haunts Norfolk Island even now: it became a penal colony between 1788 and 1814 before it was abandoned.
Stories and relics from that first colony, when the population of this topographically challenging island reached over 1000 souls, are few and mostly the provenance of historical texts such as Merval Hoare's fine history, Norfolk Island 1774-1998.
It was the second penal colony - established in 1825 for the worst felons and recidivists - which soaked the island in blood and makes for its fascinating story. Convict uprisings; torture; ruthless commandants; a population of soldiers, their wives and children stranded amidst murderers and literal cut-throats ...
From the fine homes on Quality Row (now restored) the prison was but a few minutes away. We must imagine families in their finery living in constant fear that the soldiers at their gate could be overpowered quickly if there were a breakout of convicts.
And down at the ruin of the prison it is chilling to walk between the stones, especially after having read stories of prisoners being flogged, tried lifting the leg manacles at the museum, or read of prisoners being left in cells which filled with water as the tides came in so they gasped for air and rescue.
This is the dark history of Norfolk Island which has been well documented and - despite theatrical re-enactments for tourists - remains compelling even today. The stones cry out their stories, which are reflected in place names, like Bloody Bridge where the convicts killed their overseer and hid his body behind the rocks, only for his blood to seep through the mortar and reveal their crime.
Norfolk today - so crime-free that people leave their cars and doors unlocked, their surfboards on the beach overnight - is a long way from those times. Yet people who come here for something other than relaxing days at picturesque Emily Bay, the duty-free liquor or the frequent festivals (writers, opera, jazz, country music and so on) the island offers will discover a place, as Blackburn wrote of St Helena, "soaked to the bone with the lives of the people who were once here" and an island with "a feeling of kinship that stretches back to everyone who has ever stood on this little platform".
The literal kinship between many Norfolk Islanders is as descendants of the Bounty mutineers. They came here from Pitcairn Island in 1856 after the second penal colony had been abandoned and the survivors taken back to Australia.
The same names crop up: Christian, Adams, Young, Quintal, McCoy, Buffett ... So many of the same that the phone directory includes nicknames to identify locals.
The Bounty connection is the thread that pulls this small community together. It is not uncommon for someone to be a direct descendant of Fletcher Christian, or related to him. At times it seems everyone is related to everyone else, which might explain why crime is almost non-existent.
So almost inevitably the casual visitor is pulled away from the gardens and parks, or the cafes in Burnt Pine Township, and into the history of Norfolk.
Travelling around by car - which usually comes as part of the accommodation package - allows you to explore places such as where Cook is believed to have landed, to make it to the highest point (Mt Pitt at 360m) and survey the infinite horizon, or go to one of the many museums, historic cottages or sites around this island which is rather larger in reality than the dimensions suggest.
History breathes on Norfolk, the land is rich and diverse in plant and bird life, and there is also a quaintness that is appealing to the visitor weary of cities and haste. People in cars wave at each other with a gentle raise of a finger, just as they used to in New Zealand half a century ago on country roads, and still do on the West Coast.
Here is a place where you give way to ambling cows, meet the same people repeatedly, and can swim in transparent seawater with tropical fish.
If not much great excitement happens you can still fill your days and nights easily: lunch at historic Branka House is a must-do; restaurants such as Dinos and Garrisons are excellent; you can go horse riding, fishing, take long walks ...
But inevitably I was drawn back to the long-gone figures whose lives haunt this island and make for its unique character. For day after day I walked through local history and at night read about it.
Norfolk - its people, the land itself, and the history it is soaked in - utterly seduced me. I even tried to fathom its language, a blend of Georgian seafarers' English and Tahitian, I was told.
At the end of a week we are back at the airport heading for home. I see Richard again and his cynicism has been melted by his time on Norfolk.
"I think this place is hugely underrated," he says with the glazed gaze of the newly converted.
"It's a fascinating place. I'd recommend it ... and it's so close to home. Did you go to ..."
He was right, but I wasn't really listening. I was thinking that if, like Napoleon, you were forced to live on an island somewhere ...
CHECKLIST
Getting there: Air New Zealand has twice weekly flights to Norfolk.
Further information: See norfolkisland.com.au or travelcentre.nf.
Graham Reid travelled to Norfolk Island courtesy of Air New Zealand and Norfolk Island Tourism.
