There are about 9000 indigenous Southern African plants in the garden so it's a perfect introduction to the continent's flora. When we were there the proteas were bursting into flower and tiny iridescent bee-eaters were zipping among the foliage.
As the garden has been the scene of human habitation - in the modern era alone since the mid 17th century - there are some intriguing historical features too, such as the camphor tree avenue planted by Rhodes.
Totally fascinating are the remnants of a wild almond hedge planted in 1660 by a group of shipwrecked French refugees (they'd been on their way to Madagascar) who were employed by the Dutch to form a barrier to protect its new colony.
Already running late, we left the garden with good intentions of zipping along the highway but the blindingly white sands of False Bay east of Cape Town lured us into another detour - this time to take the coastal route R44 via Kleinmond to Hermanus.
Here, proteas and other endemic vegetation were growing wild on sun-drenched precipitously steep hillsides that plunged into the Atlantic Ocean.
There were few other vehicles and most of the coastline was wild and uninhabited (what the touring route pamphlets don't tell you is that a lot of the Garden Route proper is very built up with "attractions", accommodation and shopping malls).
It was a stunning drive with views back across the bay to the Cape of Good Hope.
Eventually we did reach our destination of Hermanus, the continent's self-proclaimed whale-watching capital.
We'd almost certainly see whales here, we were told, but I was dubious - I've been told this before about whales, dolphins, kiwis...
I had to eat my words however, when, as we were checking in to our clifftop bed and breakfast lodge, I spotted a long, black shape in the waves.
"Is that ...?"
The receptionist turned round casually.
"Yes, that's a southern right... there are at least three out there today," she said, turning back to finish off our paperwork.
We were out on the cliffs in record time and sure enough, only about 10-20 metres offshore there were the whales. We counted at least five over the next few hours - some drifted in so close to us that we could hear the air rushing out their blowholes.
The whales rolled in the waves, the occasional fin waving in the warm evening air and submerged lazily leaving their tail fins silhouetted against the sky. By the time we dragged ourselves away it was almost dark.
Next day, we once again detoured - we should have been heading inland to the highway through Swellendam but instead we were somehow en route to the southernmost tip of the African continent.
Although the Cape of Good Hope is often mistakenly given this honour, it is in fact Cape Agulhas, 170km south-east of its better-known neighbour.
The road south to Cape Agulhas is a sinuous one through fertile farmland that was ablaze with fields of brilliant yellow mustard seed flowers. Small towns along the route featured that still disturbing juxtaposition of Afrikaans-style houses in the centre and ramshackle black townships on the outskirts or perched on ridges above.
According to the latest Lonely Planet, Cape Agulhas "isn't especially impressive".
Clearly the writer didn't have a geographical bent - the cape, on a cloudless still day is starkly beautiful not to mention its significance as the place where the Atlantic meets the Indian Ocean.
A monument sits on the rock-strewn shoreline that is reached along a boardwalk, beneath which were flowering yellow and blue wild flowers.
The waters of two of the world's great oceans met in a tumult of waves offshore but washed ashore crystal-clear.
A few kilometres back up the coast is the red and white-striped Cape Agulhas lighthouse. Although the sea was surprisingly docile the day we visited, waves up to 30 metres high offshore are not uncommon, making this coastline treacherous for shipping, especially in the days of sail. The lighthouse began operating in 1849 and was in service until 1968. Its light was powered in the early days by the fat from sheep tails.
Today there's a small museum inside the lighthouse and it's something of a white-knuckle climb to the top via an extremely steep ladder. The panorama from the top is almost worth the effort - even for someone like myself, who's not fond of heights.
While I was negotiating the climb, Derek discovered that the restaurant on the ground floor was offering a three-course meal with wine for about NZ$40 for two. We sat outside with a view of the sea.
It was the ex-lighthouse-keeper who served us fish that had been caught close-by that morning. Pudding was creamy, studded with sultanas and with an unpronounceable Afrikaans name.
We were replete in the sun at the tip of Africa, feeling impressed - and not just to spite Lonely Planet.
