But the few French who did stay on have left their mark - narrow streets with names like Rue Croix, Rue Jolie and Rue Viard, walnut trees and roses - and some je ne sais quois.
The French mark is everywhere: on the headstones at the beautiful cemetery surrounded by the Garden of Tane native bush on the hill - Le Lievre, Bernard, David - many of whom died far too young; at restaurants called Ma Maison and C'est la Vie; and shown by the vintage Citroens parked outside the Garthowen Bed and Breakfast, a beautiful replica of an early-1900s boarding house.
Akaroa has been known as the Riviera of Canterbury. But when we arrive at Robinson's Bay that's not our first impression. The lush surrounding hills descending to the fiord harbour where ducks float on a sea of tranquillity gives one message - relax.
Which is what we had in mind as we opened the gate to the flowering courtyard of Briar Cottage, an 1860s colonial homestead. It felt like the South of France, even more so in the spacious farmhouse kitchen where a fresh loaf of bread was on the long wooden table.
Much as we wanted to linger and enjoy this ambience, plus the cheese, chutney and lovely riesling in the fridge, we had time for just a quick taste because we were running late for our next appointment - the Giant's House.
Josie Martin, the owner and creator of this extraordinary mosaic sculpture garden, has blue hair. She meets us at the top of the outside stairs to Linton, the grandly proportioned 1880s house which she runs as a bed and breakfast and where she holds, from time to time, elegant soirees and concerts.
We are greeted by jazz music coming from a grand piano in the middle of the garden. It's Josie's latest work in progress to add to the multitude of mosaic sculptures that adorn her special piece of hillside.
Think Salvador Dali meets Mexican fiesta meets Hieronymus Bosch, with native wood pigeons passing through and you might get some idea.
Josie shows us the first piece of concrete where she started playing with mosaic bits and pieces in about 1993.
From then it just wouldn't stop - "surreal, biomorphic entities, whimsical and flamboyant, organic and eccentric" - just keep on growing.
South Island surrealism gives you an appetite. That evening we dine at Ma Maison. The evening is a little damp so we can't enjoy the outside courtyard overlooking Dailey's wharf and the Akaroa Harbour waterfront which, if the weather had been a bit kinder, would have been spectacular.
But chef Rod Parkinson's food is tres bon and we can thoroughly recommend the way he cooks blue cod and Akaroa salmon.
For a small, French town, the portion sizes are pure Mainland. Nevertheless the citrus trilogy - organic lemon and lime panacota, mango passionfruit tart, with a green apple sorbet - is too good to resist.
The next day we're up and away by 8am but not before we've had a hearty breakfast of scrambled eggs, as one must in South of France farmhouse.
I've also taken a few moments to scramble up the hill paddock at the back of the house to take in the Robinson's Bay vista.
At Akaroa's main wharf we don full body wetsuits, boots, goggles and snorkel and head out into the harbour on one of the Black Cat Group boats.
I must admit I'm highly sceptical about swimming with the dolphins. I imagined we would spend most of our time looking for them and very little swimming. But within five minutes of leaving the wharf we have a pod of four around the boat.
"Yes, they look friendly and interested," says the skipper. "Everyone in." For the next hour, we do, in fact swim, float - the wetsuits are very buoyant - and frolic with these Hector's dolphins.
They're the small variety of dolphin, incredibly curious and seem to make a special effort to check out each swimmer, flashing by so close you want to reach out and pat them. But no touching allowed.
The only downside is Akaroa harbour's naturally murky water which is how the dolphins like it, but makes visibility with the mask quite limited. Fortunately, the dolphins oblige us by regularly leaping out of the water, causing a chorus of exclamations of delight in several languages. There's hot chocolate on board after the swim and we return to the wharf elated.
Next stop is the Akaroa Cine Cafe - two deluxe cinema theatres with luxurious seating, where you can watch quality movies in comfort with a glass of wine - quite unexpected in a town that has only 500 permanent residents. The owners, Jim and Maxine Marron, have great stories to tell too. He once built sound studios for rock stars, including Jimi Hendrix.
After a delicious lunch at the cafe we're on the road again - on Paul Le Lievre's Akaroa Seal Colony Safari. Paul takes us across his farm in a four-wheel drive minivan to the ruggedly beautiful outer coast of Bank's Peninsula.
It's a very hard edge where land and water clash. But the unrelenting forces have come to some sort of agreement to create rock pools, swaying kelp and swirling surf, a harsh habitat that suits the seals just fine. Paul gives us walking sticks to help us to pick our way along the rocky face.
As he points out, it's a bit smelly, but that's the way seals are.
Seal pups are everywhere - seemingly abandoned - but as Paul explains this is how they're raised, with the mother leaving them for long periods while she goes out to sea to feed.
Seals au natural is the best way to describe this tour - nothing fancy - but special when you have guide who has such affinity with both these animals and the land. Back at Briar Cottage we reflect on the Akaroa adventure so far - action, wildlife, fine food, excellent company and plenty of je ne sais quois. And there's still more to come.
CHECKLIST
Getting there: Air New Zealand has regular flights between Auckland and Christchurch, with fares starting at $99 one way.
Where to stay: Briar Cottage.
Where to eat: Ma Maison and Cine Cafe.
What to do:
• Find out about swimming with the dolphins at blackcat.co.nz.
• Information on the Seal Colony Safari is at sealtours.co.nz.
• Visit The Giant's House.
Further information: See akaroa.com or call 03 304 8600.
Chris Barton travelled to Akaroa courtesy of Air New Zealand and Destination Akaroa.
