A spokeswoman for Mr Tremain said the Inland Revenue Department, which requires a $1.5 billion IT upgrade, would not be part of the pilot programme.
However, the cloud computing industry remains in its infancy and has come in for criticism over data security.
Apple co-founder Steve Wozniak recently warned there would be "a lot of horrible problems" over coming years as a result of businesses and agencies placing their data on the cloud.
"The more we transfer everything onto the web, onto the cloud, the less we're going to have control over it."
The Office of the Privacy Commissioner told the Herald it was in discussions with the agencies developing the Government's approach to cloud computing.
Ms Shroff said though the use of cloud services for personal information had many benefits, "it's not risk-free".
She said the government agency that put client information on the cloud would remain responsible for it, but would have to rely on guarantees about security from service providers.
"You're trusting someone else to keep your stuff secure.
"What happens if there's a data breach - eg, the server gets hacked? Will the cloud provider tell you about it so you can notify your clients?"
Ms Shroff said that if the information was stored abroad, it might be subject to other countries' laws and foreign governments might be able to demand access to it. Providers of cloud computing services could use information for their own benefit or even contract out services to third parties.
Mr Tremain said the Government was taking a conservative approach by using only New Zealand-based providers until it understood the industry and security risks better.
He told BusinessDay that Internal Affairs would work with the National Cyber Policy Office and the Government Communications and Security Bureau to develop guidelines for "offshoring government data" by the end of the year.
THE CLOUD
* Information and services are be stored "in the cloud" - on remote equipment - and accessed only when needed.
* This means less need for expensive on-site storage and IT infrastructure.
* Information stored "in the cloud" can be on computer equipment in different cities, countries and continents from the owner.
* Civil liberties groups fear personal information in the "cloud" could be accessed by overseas agencies, and be prone to theft.