Knowledge Basket kills off its spider

By ADAM GIFFORD

Online archiving company The Knowledge Basket is squashing its Tarantula research spider because it cannot afford to keep the free service operating.

"It would cost about $100,000 a year to keep Tarantula going properly," said director Dave Keet. "As it is, about 30 per cent of what we do is unpaid public good work. We had to make a decision about which bits we should keep doing."

The Knowledge Basket has free archives of Hansard, Government legislation, Waitangi Tribunal reports and Refugee Status Appeal Authority decisions. It is putting online the National Museum monographs written last century by Elsdon Best, and has a free portal for basic legal information.

It sells access to searchable databases, including law judgments and commentary, newspapers and magazines.

Tarantula, which will close at the end of the month, uses spider software from United States company Verity, which sells commercially for about $140,000.

Tarantula crawls the New Zealand web under the direction of a human editor, who identifies sites it should index.

The software can index multiple document types, including Adobe Acrobat PDF, Microsoft Word and XML files.

Unlike some spiders which search the first few hundred words of a page, Tarantula indexes the whole document text.

Keet said the software indexed sites weekly, so it was more comprehensive and up to date than tools like Google.

The first version of the site, launched about four years ago, included only law-related documents. About two years ago searches were added for material on Maori, science and agriculture, and education.

Otago University law librarian Alan Edwards said the loss of Tarantula meant it would be harder to find New Zealand-related material.

"We have promoted it to our students as an ideal way to identify New Zealand legal material on the web," he said.

"Because Tarantula worked by identifying sites likely to have New Zealand material, it meant users didn't have to think so carefully about their search terms and devise searches to exclude other material."

The bulk of New Zealand legal material on the web came from Government or law firm websites.

"Going direct to those sites and looking around for material you think should be there can be very unproductive," Edwards said.

The Knowledge Basket

Tarantula