And yet it is easily reached by air from JFK (as well as from Boston, or by ferry from Hyannis on Cape Cod), an accessibility that means that from June to September droves of Manhattanites arrive seeking a breezy contrast to the muggy metropolis.
In fact, if you take a map and draw a straight line from Wall Street, east to the Hamptons, out through the tip of Long Island at Montauk, and across 154 final kilometres of water, your pencil will pull into Nantucket Town - the pretty "capital", where I am ordering breakfast.
I opt for the Nantucketer omelette with its rich chunks of lobster and sides of toast and fried potatoes - reasonably priced at US$18 (NZ$22).
While I wait, I idly wonder whether Mr Angry, now back in full Nasdaq mode, will choose the Billionaire Benedict, a flight of fancy that - with eggs poached in champagne, two 4oz filets mignons, lobster claw, caviar, crème fraîche, pancakes and muffin - is a less affordable US$100.
He doesn't, joining me in the Nantucketer. Perhaps he's saving for something more expensive.
Perusing the property section of the local Inquirer and Mirror, I spot an advert for "Swain's Neck", a nine-bedroom, 70-acre pile at Polpis, in the northeast of the isle, that is on the market for US$59 million. This former home of an electronics tycoon is the costliest property ever to go on sale in the six states of New England.
There is no disguising Nantucket's wealth. Outside the Grill, Easton Street is a phalanx of genteel houses. Each wears a grey uniform of slatted cedar tiles - a design acknowledged around the island.
At first sight, these buildings are unremarkable. But these are multimillion-dollar residences masquerading as fishermen's dwellings. Even the White Elephant, the hotel to which Brant Point Grill is attached (and my hideaway for two nights), plays a dissembling game. It owes its name to the purchase of a cluster of cottages in 1917 by Elizabeth Ludwig, a would-be hotelier whose friends mocked her. It proved a brilliant investment.
But to suggest that Nantucket is nothing more than a bejewelled extension of Fifth Avenue is to do it a considerable disservice. For while, in places, the vibe might be Upper East Side, the overall appearance is defiantly New England.
This is noticeably so in town, a picturesque pocket that arranges itself into that idealised version of Massachusetts which haunts a million postcards. Pavements of red brick run before homes crafted from the same material (the historic structures in Nantucket Town being the only buildings to escape the grey-tile dress code). And jumbled cobblestones supply a tricky, uneven surface to the likes of Main Street and Broad Street - to the obvious discomfort of the low-slung sports cars (and the supreme indifference of the huge four-wheel-drives that ride this roller coaster with ease).
Within this little grid, a more innocent age lingers. Aside from a Ralph Lauren outlet, the deliberately archaic storefronts spurn the idea of big-brand status.
On Main Street, the Nantucket Pharmacy has a Fifties-style counter and circular metal stools reserved for those who want to pause for ice-cream floats. Quidley & Co sells marine-themed oil paintings, red sails hovering on blue horizons. And The Lion's Paw does furnishings with a nautical twist.
At the east end of Federal Street, a huge board is festooned with dead staples and adverts for yard sales, fundraisers and dinner dances - a low-tech relic of a time before Facebook. And on Orange Street, the Second Congregational Meeting House watches over everything, pale tower and dark interior a fixture here since 1809, doling out fire and brimstone to keep settler souls warm on this lonely outcrop at the edge of the New World.
Seeking less pious enlightenment, I enter The Club Car, a train carriage preserved just off Main Street. A leftover from the Nantucket Railroad, which ran from here to Siasconset (in the southeast of the island) between 1881 and 1917, this evocative antique has a new life as a bar. Where the hot chug of steam motion once sounded, now it is the clink of wine bottles as guests sup glasses of Californian pinot noir.
Places to refresh or refuel are dotted across town. American Seasons treads a gourmet path with a menu, heavy on local ingredients, that includes tobacco-rubbed duck with ginger rhubarb vinaigrette.
Elsewhere, Brotherhood of Thieves keeps things merrily stodgy, the Big Ack landing on my plate as a crab burger loaded with parmesan and bacon. Above the door, a sign declares the eatery "An 1840s Whaling Bar".
It is a reminder that Nantucket's ways were not always so refined - 200 years ago, it would have hummed with sweat and stench as a centreground of the 19th century's most brutal and yet oddly brave industries. Whaling sent Nantucket's men to distant oceans, and made the money that's still visible in the nobility of its harbour mansions.
Herman Melville even salutes the island's skill with a harpoon in Moby Dick.
"The Nantucketer, he alone resides and rests on the sea," he writes.
"There is his home."
This blood-and-blubber element of the past is dissected at the Whaling Museum. There is context aplenty: explanation that the word "Nantucket" - a verbal relic left by the Wampanoag people who lived here before European takeover - means "faraway place"; a 14m sperm whale skeleton, washed ashore in 1998, that illustrates the size of every whaler's task; artefacts salvaged from the fast-sinking Essex, a Nantucket ship rammed and downed by a whale in the Pacific in 1820 (the survival of whose crew was inspiration for Moby Dick); examples of scrimshaw - art etched on to whalebone by mariners at sea.
These voyagers set sail from an island that was windswept and underdeveloped. In many ways, it still is. Flee the town and you find a landscape of unexpectedly wild contours, long grass swaying, leafy zones - where rabbits hop and deer dart - fringing the roadside.
Nowhere is this untamed ambience more apparent than at Great Point. The northernmost tip of the island gesticulates towards Cape Cod. To reach it requires effort: turn left on Wauwinet Road in the island's northeast (where The Wauwinet hotel does soft elegance, fine dining and a waterside lawn lush enough for a Gatsby party), then a five-mile trek along dunes (feasible on foot, but a hired 4x4 helps). Here, wary gulls protect their nests and fat seals loll on the shingle. A squat lighthouse, now largely ornamental, hints at the treachery of the offshore currents.
The rest of the island is laced with cycle paths, and perfect for exploration on two wheels. (Most hotels offer bike hire.)
On a warm Saturday, I pedal to the east, gliding down Polpis Road, past the Sankaty lighthouse, flashing its traditional warning through the sudden mist that swaddles the adjacent golf course.
A little less than 2km on, Siasconset is quiet and lovely, a village braced against the ocean. And though a handful of stores lurk here - an unabashedly quaint post office, a grocery, Claudette's, a cafe - the atmosphere is sleepy.
The houses sport signs that read "Low Tide", "Ensconced" and "Treasure Chest", and the lane along the front makes me smile, surely ranking as the least busy "Broadway" on the planet.
Further along the south shore, Surfside is Siasconset's tiny cousin, aptly named in the breakers that crash upon its sand. It takes perhaps 45 minutes to cycle here from the east of the island, another 45 to foray to the west.
As I flit along Madaket Road, the scenery becomes wilder. Dionis Beach, on the north, is flat, spartan and deserted, the sea enveloping the land in a chill, sullen embrace. And two miles along Eel Point Road, this narrow route dispenses with surface, turning to dirt, dust and wide puddles.
Weather-beaten and desolate, the extreme west of the island could not be more removed from the sophistication of the town, ebbing to a close in two distinct spurs.
Together, Eel Point and Smiths Point might be an open lobster claw, gesturing towards New York. And New York nods back, shouts that it will be across later.
And this year, it might try to relax.
- INDEPENDENT
