When they took over Terraza in 2007 the industry was huge and there were about 40 other growers in Hawke's Bay.
Potts said one reason for the decline in the industry is the fact that saffron growing, picking and processing is very labour intensive which put a lot of people off.
"When we started there were multiple growers and brands here, now there are probably only four or five around the country," she said.
"The romantic part of it is the attraction, then reality hits and people just walk away.
"They need to be realistic, people can't think they will buy 10,000 corms and in the next few years make their first million.
"The money isn't as much as people would think and people don't have to invest a huge amount into it so they feel it's easy to walk away."
Tyro said another reason for the decline was that the climate in Hawke's Bay hadn't been ideal for the last few years.
"It's a lot more humid than it used to be which makes the saffron more prone to disease.
"Climate has had a huge impact on us.
"The number of growers in Hawke's Bay and around the country has dropped dramatically over the last five years."
When Potts and Tyro started growing saffron in 2002 it was just taking off in Hawke's Bay due to Terraza Saffron's previous owners Ian and Rochelle Schofield's promotion of the industry in Hawke's Bay and encouragement of new growers, Ms Potts said.
Tyro and Potts grow saffron on their 1.4ha in Maraekakaho but also use a network of growers around the country, to mitigate climate risk, for Terraza Saffron.
Consumers would not be able to tell the difference between Terraza saffron grown in Hawke's Bay, Central Otago or Canterbury because they are all quality-controlled, grown and processed under the same specifications.
This season the couple will be mainly relying on their growers because Hawke's Bay has again been too humid, Potts said.
They expect to harvest about 200g from their plot of land later this month and will be getting between 3.5 and 5kg in total from their network.
To make 5kg a total of 750,000 flowers would need to be picked and processed. Flowers need to be picked the day they bloom.
This season they will probably only have enough to supply their customer-base of fine food outlets and chefs.
In the past, Terraza Saffron was able to sell to the public but this has not been possible for the past few years.
"We're in a position where demand exceeds supply, we always want more," Ms Potts said.
While this may all sound a bit doom and gloom, all hope is not lost, she said.
"We're in the process of rebuilding."
During their best years from 2008 to about 2012 the owners sold a lot of their corms to help other people get started in the industry and to spread out the growers.
However, they have lost 90 per cent of those growers.
Tyro and Potts are now trying to rebuild their crop and would eventually like to start promoting the industry again.
"I'd like to get back to the volumes that we were doing in our heyday," Mr Tyro said.
Even if Hawke's Bay continues to be too humid the couple will not give up on saffron.
They had even thought of the possibility of growing elsewhere if all else failed.
"We will not walk away from it, it's part of us," Potts said.
Their love affair with saffron began because it was something they could do with their children.
"The kids come out and pick with us and we all process around the table together," she said.
"I love going out in the morning, seeing and smelling the flowers.
"I never used saffron before we started growing, now I use it weekly."
Tyro said it was also really nice to be involved with something from the very beginning to the very end.
Especially something that has such a good reputation and high quality, he said.
"I would certainly put our saffron up with saffron grown anywhere in the world.
"We've been told by chefs in the past that our saffron is far more intense than anything they've come across."
He said this came down to the care and attention to detail of the growers as well as the drying process, the land, the climate and the freshness.