This allowed evidence to better inform policy decisions in areas where often evidence had often been lacking.
"We are using insights from cutting edge social and behavioural sciences to underpin the powerful new tool of citizen-based analytics.
"This gives us the potential to gain unprecedented understanding of what interventions could work best, with whom and in what context.
"This is the essence of scientific advice to inform policy and there is a lot of interest globally in what New Zealand is doing.
"It can be a game changing tool for governments seeking to better support citizens' needs."
But it wasn't straightforward - and there were areas where policymakers needed to take care in drawing on the vast amount of data available to them through pools such as New Zealand's Integrated Data Infrastructure (IDI), which contains microdata about people and households.
"Firstly there needs to be broad transparency and understanding regarding the intervention logic and the consequential use of scientific underpinnings to inform data analytics in social policy making," he wrote.
"Secondly there needs to be attention paid to the governance, management and stewardship of data and clarity and accountability as to purpose of collection and use."
Any databases to be used together needed a common set of technical standards and definitions of metrics, a standard format for packaging the data and a uniform process across every agency or data-collection point for data cleaning.
Care was needed when using third party sourced data - and also around potential data misinterpretation and algorithmic bias.
While there was value in creating as full a data picture as possible, which included data from both within and beyond governments, there had to be clear boundaries as to uses that relied on identifiable data, versus anonymised data.
"A further set of issues exists when data are used to predict outcomes at an individual level," he said.
"While prediction based on risk factors is a key objective of citizen-based analytics to assist targeting interventions, it needs to be emphasised that in general such predictive approaches will identify risk and resilience factors based on group characteristics and there are significant limits and dangers in extrapolating this to a specific individual."
There also needed to be an "understanding" of the relationship between citizen-based analytics and other aspects of social policy development and implementation, and a "culture change" at every level, stretching from clients and service providers to policymakers and politicians.
Sir Peter said the new approaches would require governments to be willing to accept findings which could be "inconvenient" in political or ideological terms, because they would suggest the value of shifting resources around the system as social insights were developed on a robust basis.
"Only then can these important developments in understanding the lives of citizens translate into more effective policies that could benefit the whole of society."
New Zealand Science Media Centre director Peter Griffin said the paper reflected what had been a big focus of Sir Peter's work in the top role.
"This paper talks a lot about social licence, which symbolises one of the key planks of Sir Peter's agenda as chief science advisor to date - urging science to earn the social license to do its work."
