Mrs Clarke herself is long gone. She took on her husband's bakery and confectionery business when he died in 1880, moving to this two-storey premises in 1891, then described in The Western Star as a handsome addition to the architecture of the town. Now it's just one among a scattering of businesses. We get our latte, and some delectable home baking, too - "date scones like my mother used to make", says my companion - while seated among charmingly mismatched furnishings, paintings by local artist Chris Flavell, and an eclectic mix of folk, who wander in and out for their early-morning treats.
It's now we realise there's more to this area than meets the eye - and we've already weeded the Caitlins and Stewart Island from our itinerary, for both require more time than we have to play with.
Should we take in the arts and crafts trail, or the bush walks? Historic sites such as Templeton's restored flax mill, Maori settlements, or the Thornbury Vintage Tractor Museum? We settle on the Te Hikoi Cultural Heritage Museum to start with, expecting an array of dusty cases and mismatched labels, a few fish hooks and perhaps some old whaling yarns.
What we get instead is a delightful stroll through the region's colourful and extensive history. This was the second area to be settled in New Zealand after the Bay of Islands, as we discover on being ushered into the bowels of a wooden sailing ship, the decking and gunwales fitted out by a local boat builder.
There, we watch Riverton's history unfold as told by an old seadog to his granddaughter, complete with riveting re-enactments. He tells of folk like Jacky Price, stranded on inhospitable Solander Island with his wife, Hineawhitia. They built a sealskin coracle and paddled 38km of open sea to Preservation Inlet. That feat seems even more astonishing when we see the coracle built by the film-makers, little more than a bathtub clad in deer instead of seal hide, and now part of the museum's exhibits. Go to sea in that? Those folk were clearly made of sterner stuff. Mind you, so are their descendants.
This museum was built and kitted out after locals - and there aren't that many of them - raised the money to build it, and it's a wonderful mix of entertaining and informative, with interactive displays enlivening the more traditional exhibits.
Riverton's history starts in Polynesia, with the early South Pacific settlers naming the area Aparima - and telling of a giant whale, eating through the land to form Foveaux Strait. Three southern Maori tribes - Waitaha, Ngati Mamoe and Ngai Tahu - set root here, centuries before the arrival of Abel Tasman and James Cook. Soon others followed, attracted by the plentiful timber and the safe harbour from which to chase fish, whales and seals. Among them, captain John Howell established a whaling station in 1836, married a high-ranking Maori woman - Khikohi - and opened the first school. It was a hard life back then, brought home to us by the museum's mud-floored hut; a whaler sleeping off his beer by the fire while his daughter skins a rat and his wife works over the washtub. Weta Productions must have had a bit of fun with this commission.
A fragment of the original forest hovers above Riverton, flourishing as Aparima Pest Busters beat back encroaching rats, stoats, possums and magpies. But we don't feel like bush walks. Instead we potter down the rocks highway, a narrow road which meanders past beaches and bays, fringed with baches and houses looking out towards Stewart Island and beyond it, the Antarctic.
This must be a stark coast in winter; today it's as picturesque as Riverton's harbour, with clusters of fishing boats. It's too early in the day for swimmers, but folk are already fishing and these shell-strewn beaches are popular in summer, with quirky play-parks if the kids need a change; one features rough-hewn wooden whales and seals to clamber over.
However, it's time to head for Colac Bay and our rendezvous with the giant surfer. The small beachfront community hosts swimmers and surfers, the tavern - with its tent sites and internet access - no doubt doing a roaring mid-summer trade. It's gorgeous, still quiet but a long way from the early days when, in 1838, John Jones bought this land for £25, 48 spades, 20 pairs of blankets, 50 pea jackets, 48 flushing trousers, 120 shirts, 24 pairs of shoes, a cask of gunpowder and a keg of tobacco.
We divert to Isobel's weaving studio. Weaving is a historic craft - but not as historic as the rock down here. We'd seen some in the museum - argillite, and 260 million years old, and the region's complex mineral footprint with its 120,000-year-old sandstones, its fossils and ancient lava flows deliver an evocative link to Gondwana land.
But forget history, we're hoping to make our fortune on gemstone beach. We're after stones with a waxy feel; we're after the Jasper and quartz, garnet sand and sapphires, but there are too many colourful sea-rounded pebbles to remember our instructions and soon we have pockets-full.
We haven't made our fortune, but who cares when they're such pretty colours? I later find a flat green memento in the rental car, a reminder that I'll have to come back. After all, the rest of Southland beckons.
