Library captures digital heritage

Like a rapidly deteriorating Alzheimers patient, New Zealand is losing its recent memory.

Penny Carnaby, the National Library's chief librarian, says much of our history could be lost because of the digital age in which we live.

In pre-internet times, most of our heritage was printed, painted or recorded. But now, a lot of the material that future historians will want to study to give them a sense of what New Zealand was like in 2005 is online.

A lot of information and commentary on recent events that New Zealanders have felt enormously proud of - The Lord of the Rings trilogy or the America's Cup - is on the web and not in print, says Carnaby.

"If you did not preserve that we would not have the story."

Or to put it another way, somewhere out there the next Janet Frame may be bashing out brilliant emails. But while these will hang round on a few computers and servers for a while, much of it will simply disappear.

Carnaby and the library's innovation and technology services director, Graham Coe, are part of a team working to preserve our digital heritage.

Last year, in a move supported by all political parties, the Government committed $24 million over four years to the project.

Not even Act leader Rodney Hide objected to this use of taxpayers' money - "because we'll be preserving his speeches as well", says Carnaby.

The digital archiving project is a foundation stone for New Zealand's Digital Strategy, released by the Government in draft form last year and due to be finalised by the middle of this year.

The project will see video, sound, text and graphics saved as digital objects.

Websites or blogs that have existed for a few years will also be saved.

Because of the earthquake risk in Wellington and to protect our history from future cyber-vandals, the archive will be replicated on computers in Auckland.

"Security will need to be like Fort Knox," says Carnaby.

The project will be completed by 2007. After that it will need to be maintained and added to, and the National Library will also start digitising its physical collection.

Coe says not everything will be saved and this raises some interesting questions about what to include and what to leave out.

Most hard-copy New Zealand publications are already deposited with the National Library so these will also be saved electronically.

Carnaby says even some give-away advertorial publications will be preserved as social ephemera of our era.

But what about local porn or hate sites? Saving this type of material may irk social and cultural conservatives, but leaving it out would effectively airbrush our history.

Coe says museum curators and archivists make these types of judgments every day.

Journalists, too, edit in a similar way as they consciously or unconsciously decide what to leave out of their stories.

On the technical front, Coe says the library is about three months away from choosing a technology provider. This may be an IT company or an electronic publisher.

The hardware and software could come from different suppliers.

The National Library now uses Sun Microsystems technology, which it is happy with. Coe says the project's RFI (request for information) mentions the library's relationship with Sun but leaves the door open for other providers.

The manager of the National Library's innovation centre, Steve Knight, is internationally recognised for his work in developing meta-data systems and is the intellectual leader of the project.

Together with what Carnaby describes as some "seriously bright technical guys", Knight is developing a meta-data preservation tool.

Because technology changes so fast, the digital archive will need to be transferred to a new operating system every 10 to 15 years. But it was important to preserve the technological context in which a poem, for example, was created, said Coe.

"It's not just the words but how they are arranged on the page which gives a poem its cadence and meaning."

What Knight's meta-data preservation tool will do is allow a poem written using Microsoft Word software to be accurately replicated 100 years from now, even though no one will use Microsoft Word then.

"It does not sound that sexy but it's the key to preserving digital objects," says Coe.

Carnaby and Coe are also excited about the international collaboration happening around the project and the fact that New Zealand is leading the world in preserving its digital history.

If successful, it is hoped the National Library's archive will be used as a reference site for other libraries round the globe looking at similar digital storage.

The project is also being peer-reviewed by a variety of heavyweight academic institutions, including the British National Library, the Royal Library of the Netherlands, and Cornell University and the Getty Research Institute in the United States.

"We put things in front of them and they give us input," says Carnaby.

"It's not an issue that any country can solve on its own."

The bottom line is that the archive will make the jobs of historians - and journalists - much easier in the future. Imagine, with a few keystrokes, being able to check out your great-grandfather's website or your great great aunt's blog.

For the record

* The National Library's role is to collect and maintain literature and information resources relating to New Zealand and the Pacific.
* In the digital age, that means preserving material published on websites and other electronic media.
* The Government has provided the library with $24 million for a four-year project to develop systems for preserving our digital heritage.
* The library is about three months from settling on a technology platform for the project.