They had been stored at Mount Earnslaw Station for years, having initially been given to Tommy Thomson to look after.
"It's the heritage value, probably more than anything else, that's upsetting," Geoffrey Thomson said.
"It's not just any old item - it's ones you've had in the family for a long time and someone's broken in and taken them.
"The fact that it was broken into is, I guess, the biggest concern. It's not as if anything was just lying around, someone's actually quite clearly known what they were doing.
"They've broken into the cabinet and just took everything out of it."
His safe was in a sleepout area of his property and was "not obvious".
He suspected whoever took the guns knew they were there and said it was a "bit of a shock" it had happened in the tight-knit community.
"Everyone knows everyone pretty well in these small areas ... we'd dearly love whoever did it to feel a bit of a conscience and bring them back.
"That would make us pretty happy, I'd have to say."
Miller said one of her guns was a single-shot .22. She estimated it was about 50 years old and she had used it to shoot her first rabbit.
The other was a semi-automatic .22 which did not have the bolt or magazine with it.
"So they've just taken the gun.
"They haven't stolen them to use themselves, I don't believe.
"It makes me sad and angry - for Geoffrey, too ...
"The town's changed."
In a post on social media Miller called the thieves "the lowest of the low" who had no thought or regard for the memories and sentimental value of the items they had taken.
"There will be someone who knows something.
"If you are this person, grow some balls ... and be the better person," she said.
Sergeant Steve Watt, of Queenstown, said police were aware of the incident and had sent a scene of crime officer to the property to investigate.
"We ... are appealing to the public who may have seen any suspicious vehicle in the area to notify us."
