The Persians came here too - intermittently for centuries the Omanis and Persians clashed over their coastal territories and control over the Strait of Hormuz, the narrow stretch of water that links the Indian Ocean with the Persian Gulf. Ironically, today Oman has one of the more harmonious relationships with Iran of any Middle Eastern country.
One of the most stunning sights in Muscat is the Sultan Qaboos Grand Mosque which highlights those links across the Persian Gulf.
Sultan Qaboos ibn Said (whose family has been in power since 1747), is the current ruler of Oman and is revered and loved throughout the country, perhaps in part because he has funded projects such as this mosque, along with world-class highways and full government services even in small towns in the desert.
His mosque however must be his crowning achievement. It was opened in 2001 after a six-year building project and can accommodate a total of 20,000 worshippers in the main mosque, the women's prayer hall and the outdoor courtyards.
A jaw-dropping feature of the mosque is the grand chandelier, which is eight tonnes of glittering Swarovski crystals and gold-plated metal. There are also 34 other, slightly smaller, chandeliers to keep it company.
The floor of the main prayer hall is covered in another stunning piece of craftsmanship: the carpet made by 600 weavers from Iran's Khorasan province is entirely hand-knotted, and was made in situ, taking four years to complete. The beautiful work contains 1700 million knots and features 28 soft pastel colours.
Visitors are welcome in the mosque (it's one of the few places in Oman where women must wear a headscarf and no shorts for either sex are permitted but those restrictions are a small price to pay to visit) and it's a tranquil, stunningly beautiful place to explore.
The tile work, including the very intricate mosaic faience tiles, is superb and the white, smooth cool marble is a pleasure to walk on in bare feet. Fountains and pools cool the air outside.
Another place that captivated me was the Muttrah Souk. Muttrah is one of Muscat's distinctive neighbourhoods and the souk is just across the cornice (the road and promenade that hugs the coastline).
Oman has had hundreds of years of history as a trading centre for the Middle East, Africa (the Sultanate of Oman once ruled the archipelago of Zanzibar off the coast of East Africa) and India, so it's appropriate the souk has goods from far and wide.
One that is almost entirely Omani however is frankincense - a crystalline gum resin obtained from trees in southern Oman (the tree also grows in a few other places on the Arabian Peninsula). Known as the perfume of the gods and as one of the three gifts presented to the baby Jesus by the Magi it's still a much sought after perfume.
I bought a packet of the pebble-like pieces in the souk but you do need to also have a burner (they can be heated with charcoal or powered by electricity) to release the aromas. It's used not only for religious and cultural reasons but is also considered a very good clothing fumigant!
Like most Middle Eastern cities, Muscat really comes alive at night. Along with the souk, the seaside cafes along the cornice were doing a roaring trade as people sat enjoying the sea breezes, drinking tea and Arabian coffee and smoking fragrant sheesha (hubble bubble) pipes, as the sun went down.