But in an unexpected twist, the researchers also found no evidence the selection process was able to meaningfully predict the likely success of different proposals.
The application process for the fund is organised into two stages: in the first, an initial one-page proposal is reviewed by a subset of a panel of relevant scientists and given a preliminary score, and around three quarters are rejected.
In the second stage, longer proposals are submitted and sent to anonymous referees, typically internationalo ones, for review.
Applicants are then given the chance to respond to referee comments before the panel scores and ranks the proposals.
"Interestingly, we didn't find a link between a project's future success and the rankings given to it by the second-round panel," Dr Jaffe said.
"This means there is no reason to expect diminishing returns if Marsden funding were increased.
"It also means the significant resources devoted to the second round evaluation could be reduced without degrading the quality of decision-making."
The analysis, based on 1,263 Marsden proposals which reached the second stage of review between 2003 and 2008, found overall that 41 per cent of the second-round proposals were funded.
Around 25 per cent of the proposals were "Fast Start" grants, awarding funding for early-career researchers, and slightly more than half of these were funded.
The average researcher on these teams made six proposals and received 1.2 grants between 2000 and 2012.
The researchers also identified the approximately 1500 New Zealand based researchers named on these proposals and examined their annual publication and citation record between 1996-2012.
"We were very foturnate to have access to all the funded and unfunded proposals, including their evaluation scores," Dr Jaffe said.
"This meant we could control statistically for potential bias driven by the Fund's efforts to fund projects that are expected to be successful."
Marsden Fund Council chair Juliet Gerrard wasn't surprised by the study findings that all of the proposals in the second round were of very high quality, adding the council had said for many years that the number funded could be doubled without a drop in quality.".
We believe that the second round peer review by international referees and a panel of experts, is critical to maintaining rigor in the process and is the method most likely to spot proposals that have something really special," she said.
"We continually seek to improve our processes by benchmarking against other funds."
She noted the Health Research Council had trialled a lottery system in its HRC Explorer grants, which could be good to compare against the traditional peer review system it used for their other grants.
"At the moment, the confidence given by external peer review, particularly in differentiating the 'truly excellent' from the 'very good' is critical to the current success of the Marsden Fund."
Dr Nicola Gaston: Motu stocktake a 'great step' toward transparency in science funding
President of the New Zealand Association of Scientists (NZAS) Dr Nicola Gaston saw the effectiveness of the Marsden Fund as a "great step" towards transparency and the use of evidence in setting policy around science funding.
"Given the anecdotal evidence in the scientific community, in particularly that supplied by the winners of our most significant awards for scientific achievement, it is absolutely not surprising to see the positive effect that Marsden funding has on the productivity and impact of researchers."
What she did find surprising about the study, however, was a conclusion that the rankings made by the panels at the second stage of decisions did not predict the ultimate impact of the work.
"However, this is not as surprising at it may seem, given that the panels themselves are broad and they are heavily oversubscribed, meaning that sometimes panelists may be comparing apples and oranges in terms of the research topic.
"The second point made by the research team, that the resources invested in the second stage review of proposals may be largely viewed as wasted, I think is rather more nuanced.
"From the perspective of the funders this is perhaps true, but the reality is that the expected process of evaluation determines the behaviour of the researchers who apply to the fund."
For example, she said, one could imagine that if the second round were perceived to be less rigorous than it currently was, the fund would attract a larger number of less credible applications.
Dr Gaston also felt the study's suggestion that there were fewer alternatives to the Marsden Fund for the support of fundamental research in New Zealand than in the US did reinforce the primary conclusion, that the level of funding distributed via fund was simply too low.
Further, simulated trajectories in the study for publication and citation counts, under scenarios with and without Marsden funding, demonstrated one very crucial fact: the baseline slope was negative.
"Funding for research of the kind available through Marsden is not just a nice-to-have; funding for research is an absolute necessity if we are to sustain a vital and productive research community in New Zealand."
Former NZAS president Professor Shaun Hendy said the study had thrown up some "fascinating insights".
"Most importantly it shows that receiving a Marsden grant does lead to higher productivity and impact, at least in terms of papers published and the citations those papers receive.
"This does not surprise me, but it is very exciting to see the benefits of Marsden funding quantified for the first time."
Professor Hendy said the research demonstrated the benefits of the sustained collection and retention of science and innovation data, and he felt the Royal Society of New Zealand should be commended for its commitment to doing so.
"Unfortunately, the Ministry of Business, Innovation and Employment and its predecessors have done a very poor job of curating their data, meaning that much of the rest of our funding system will remain opaque for some time to come.
"The Ministry needs to put in place systems and practises that will allow these types of studies to be undertaken right across our science and innovation system."
Steven Joyce: Govt committed to further funding boosts in investigator-led science
The new study comes just ahead of the Government's release of its National Statement of Science Investment on Monday, which would provide a blueprint for measuring impact and excellence in New Zealand's science system.
Science and Innovation Minister Steven Joyce said while the Motu paper was "by no means the full story" on the Marsden Fund, it did make a positive contribution.
"It is important to note that the research relates only to Marsden Fund funding decisions made up until 2008," he told the Herald.
"Since that time there have been significant changes including a sizeable increase in the size of the fund, and the objective of the fund extended to require funded projects to be of 'long-term benefit to New Zealand'.
"The paper also doesn't seek to measure actual outputs of actual research projects funded by the Marsden."
Nevertheless, he said, the study was a positive review of the fund's benefit to New Zealand, while raising some questions about the process of assessing proposals for funding.
Mr Joyce said Marsden Fund pool had increased by 42 per cent since 2008, and he was committed to further increases in investigator-led science funding in the years ahead.
"This research helps to give the Government more confidence in the quality of additional proposals when we next have the opportunity to increase the size of the fund," he said.
"However the big shortfall in New Zealand's investment in research and development currently compared to other OECD countries is the investment made by the private sector.
"That is why we have a comprehensive range of programmes through Callaghan Innovation and the Primary Growth Partnership to help grow that private sector investment."
The Ministry of Business, Innovation and Employment had recently been going through a process to simplify the assessment of proposals for the MBIE Contestable Research Fund, which was also independently run, with funding decisions made by the MBIE Science Board.
Mr Joyce said that process, in relation to the 2016 competitive process, would be announced shortly, and he would be asking the board to share their thinking in detail with the Marsden Fund Council.
"It is important that there is a robust, independent assessment process to inform which applications are granted funding, but it needs to be as cost-effective as possible to maximise the availability of limited funds for actual research.
"Any suggestion, however, that we just fund 'everything' is simplistic as there will always be more good research proposals than there is funding available - that is true all over the world."
