So the researchers then ran a series of regressions which controlled for a list of potentially relevant variables including incomes, net wealth levels, age, gender, partnership status, home ownership, investment property ownership and ethnicity.
The aim was to compare like with like as far as possible and see, within those subsets of the survey population, if membership or non-membership of KiwiSaver made a difference to the accumulation of net wealth, and how much difference it made.
The survey was large enough - around 10,000 people - to still give statistically robust results when sliced and diced in that way.
See the full research here:
"The results of the regression analysis ... do not support the hypothesis that membership of the KiwiSaver scheme has been associated with greater net wealth accumulation amongst its members," Law and Scobie conclude.
"That is, most regressions specifications yield coefficient estimates on KiwiSaver membership that are not statistically different from zero."
Translation: All they can say about the effect of KiwiSaver on net wealth accumulation is that they can't say it was not zero.
Two potential problems for this work are that the period for which they have data covers only the first two or three years of KiwiSaver, which has now been running for seven years, and the period also straddles the global financial crisis and a recession that affected asset prices and incomes.
"However, regression estimates suggest that tenure in KiwiSaver has little effect on net wealth accumulation," Law and Scobie say.
In other words, when they excluded those who had only been members for part of the 2008 to 2010 period they still did not find a positive association between membership and wealth accumulation.
So while more data would be desirable, these data give no reason to suspect it would change the overall conclusion.
And the global financial crisis would be relevant to the question addressed by this study only if it had affected members and non-members of KiwiSaver differently for some reason.
But the obvious differences that might have an effect, like wealth and income levels, or a preference for real estate over financial assets, were controlled for in the regressions.
Law and Scobie see this work as reinforcing the conclusions of an earlier study they did in 2011 based on a Colmar Brunton survey.
"This study, which used completely different techniques and data from the first evaluation, provides a second piece of evidence which suggests that KiwiSaver membership, at least until 2010, had not been associated with greater accumulation of net wealth, and hence improved retirement income outcomes."
The study
*The Survey of Family, Income and Employment (SoFIE).
*It began in October 2002 with an original sample size of about 11,500 households, amounting to over 22,000 individuals 15 years of age and over.
*Concluded in September 2010 after running annually for a total of eight years.
