Mr Price said the man was arrested without incident after a short foot-chase.
Police said no one was injured, but the Linwood Winz office will remain closed while police speak to staff.
Earlier today all other Winz offices were placed in lockdown as a precaution for staff and the public, but have now reopened.
"Everyone has a right to be safe and feel safe whether at home or at work," Mr Price said.
"This behaviour will not be tolerated and the prompt action of Winz staff alerting police ultimately led to the man being located and arrested.
"Police will be visiting Winz offices around the district to reassure and support staff."
Mr Price said a staff member and security at Linwood Winz had a confrontation with the man, who then left the office and went to his vehicle to get the firearm.
Mr Price said he believed the firearm was a shotgun and it was not yet known whether it was loaded.
"I'm not aware of whether or not the gun was actually used ... but a gun was presented outside the building," Mr Price said.
Police would be interviewing the man and Winz staff members to try to understand what the grievance was about, he said.
Security guards and some staff members intervened to block the man back from getting back into the office.
Ministry of Social Development (MSD) chief executive Brendan Boyle said in a statement the ministry took all threats to staff very seriously.
"MSD and police will continue to work closely together. People cannot and should not be able to threaten others without consequence," he said.
The incident comes a month after the first anniversary of the fatal shooting at the Ashburton Winz office.
Peggy Noble and Leigh Cleveland were killed in the September 1, 2014 shooting on Cass St.
Russell John Tully has been charged with the murder of the two women, as well as the attempted murder of Lindy Curtis and Kim Adams.