Gunning for the answers

The information is out there, but where? FRANCIS TILL explores the options for searching the internet.

The public internet includes well over three billion pages and you want only one of them. You're sure it's out there - everything else is - but you don't know its name or where it is filed.

You could always call your good friend Jane, she knows how to find things on the internet, but maybe it's time to try finding it on your own. After all, it might be embarrassing to let even Jane know what you're looking for.

Your first challenge is selecting a search engine - or engines.

Most experienced searchers agree that, for the moment, the two best search engines for general searches are Google and Alltheweb. So why use two search engines?

Search engines don't search the web

They don't look at the internet to find answers to your questions. They look at databases they've built and indexed during their own travels around the net. All major search engines have their own databases - and almost all of them index them differently.

This means that users can get completely different answers to the same question ("query" in jargon), depending on the engine, and how the question is asked.

Google and Alltheweb have among the largest and most diverse databases available, and they're indexed well, so you have a much better chance of finding what you're looking for than on a small, badly indexed search engine such as the popular "natural language" engine Ask Jeeves.

Six of one, 32 of another

Not only are all search engines different, not all of them even do the same thing. There are thousands of search engines, and even some, called meta-searchers, that just search other search engines, which can be a valuable way of probing multiple databases quickly.

Popular meta-search engines that are easy to use include Vivisimo and Metafetcher. A valuable index of engines is at the Big Search Engine Index.

But if you are looking for something unique, one of the many thousands of special topic search engines might have what the big guns overlooked.

There are many indexes of special search engines, but you may be better off asking a general search engine to find a special search engine for you. The returns are likely to be more current, the links are more likely to work, and you won't have to comb through directories looking for ones that have indexed content of interest.

That said, Informational.com provides an extensive "directory of directories". Among the sites: a searchable index of comic book stores around the world (there are four listed for Auckland, with phone numbers, street addresses, email and store/inventory profiles; Telecom's online yellow pages lists one, with only an address and phone number) at the Master List.

Well-known - and well funded - specialised search engines include www.psychcrawler.com, www.politicalinformation.com, and www.inomics.com, all of which focus on a specific subject (psychology, political science, and economics, respectively), and all of which can return highly targeted information not generally available on the web.

Some search engines, such as Yahoo, are called "directories" and get away with having relatively small databases because they evaluate websites before they enter them into indexes. That means, in theory, that everything inside a topic area - for example, new zealand media - will be a useful resource. Unfortunately, these are often more like unedited encyclopaedias than search engines.

Still, we're only scratching the surface.

Best estimates are that no search engine has indexed more than a small fraction of the web - which is growing faster than it can be collected - let alone the internet, of which the web is only part, or the "deep web" which some special engines are beginning to explore.

Check it out

It's perhaps appropriate that libraries are among the best chroniclers and cataloguers of search engines. For a sample, try the University at Albany's library collection on searching the web.

Spelling counts

Because the English language web is dominated by American spelling, many searches can come to naught through habit-driven use of Kiwi spelling but basic spelling errors, such as mistyping "nateral" for "natural", can often skew results dramatically. If the error is obvious enough, some engines will prompt for an alternative spelling. If not, keep an online American dictionary handy.

So does syntax

Good search engines try to anticipate human errors in many ways, one of the most important being the way a query is framed. Alltheweb, in particular, will modify queries in a number of helpful ways, tell you what it did to your query syntax and give you the option of undoing its helpful intervention.

Both Google and Alltheweb, in fact, can parse most natural language queries to make a useful approximation of a "proper" query. So if you ask either a natural language question such as "how long do dogs live?" you'll find reasonably responsive results (Google is better at this), but both prefer the query syntax (dog lifespan).

Fine-tuning the phrase Remember to group phrases in double quote marks when using basic search - Google and Alltheweb will do this automatically for you when the phrase is common, but don't rely on that.

A search for your net nz returns a vastly different result from "your net" nz .

Finally ... If you're looking for New Zealand content, just add the letters nz to the end of any search query on Google or Alltheweb.

Links

Google

Alltheweb

Ask Jeeves

Vivisimo

Metafetcher

Online American dictionary

University at Albany's library

Big Search Engine Index

Informational.com

Master List

Psych Crawler

Political Information

Inomics

Search engine quiz - the obscure seven

Try your searching skills on these seven questions. Your aim is to get the answer on the first page of search results and in the number one position. Answers below with my search terms and rankings - but don't look until you've tried yourself.

1. Why was Tennyson named Lord?

2. How hot is a habanero pepper; and how does it compare with the thai?

3. How far is it from Auckland to anywhere?

4. Who was on Shortland Street in 1992?

5. Which Xena actors have made films?

6. Where and when was the last earthquake in New Zealand?

7. What are the lyrics to that 199? song Bic Runga did with that guy?

Answers

1. Alltheweb - query: Tennyson Lord.

Result: Queen Victoria was continually offering Alfred a baronetcy, which he kept turning down (he was rather shy). It was not until 1884 that he became Lord Tennyson.

Position 3: Alfred Lord Tennyson Biography.

2. Alltheweb - query: chili pepper temperature.

Result: Pepper (capsicum) temperatures are measured in Scoville Units; the jalapeno has 3500-6000 SUs; the thai has 80,000-300,000; the habanero has 300,000+.

Position 1: Capsicum; all about chili pepper plants (alltheweb.com/go/1/H/web/http/www.chili-pepper-plants.com).

3. Google - query: calculate distances.

Result: Indonesia/Bali travel website with extremely popular distance calculation tool. US cities plus 500 global.

Position 1: How Far Is It (www.indo.com/distance).

Alltheweb - same query.

Result: USDA site with instant Java calculation tool that will determine distance between any two latitude/longitude figures.

Position 3: (www.wcrl.ars.usda.gov/cec/java/lat-long.htm).

Google - related query: find latitude longitude.

Result: MIT Geographic nameserver; has over a million geographic names in database, all with co-ordinates.

Position 1: (www.mit.edu:8001/geo).

4. Alltheweb - query: "Shortland Street" cast 1992.

Result: Link to International Movie Database site listing all Shortland Street actors from beginning of show to present day, with years in which each appeared.

Position 1: (webtv.imdb.com/Title?0098911).

5. Trick question after the Shortland Street results.

Enter Xena in this specialised search engine for the cast of that show. The complete filmography for each cast member is available by clicking on individual names.

6. Google - query: earthquakes nz.

Result: Institute of Geological and Nuclear Sciences tracks seismic activity in real time; extensive archives.

Position 1: (www.gns.cri.nz/earthact/earthquakes).

7. A search in several parts: What song? When? What guy?

Google - query: Bic Runga

Result: Columbia Records official Bic Runga website with complete discography.

Position 1: (www.bicrunga.com). The song is Good Morning Baby, in 2000 with Tim Finn. Earlier recorded with Dan Wilson. (There is a soundtrack here if you want to confirm). The website also has lyrics.

Alternative search: Deja.com (defaults to the new Google Groups interface) Search term: "Good Morning Baby" "Bic Runga".

Result: Position 2 is a history of usenet postings from the beginning of internet time. Bic Runga discussion groups are on the site, along with the lyrics for this song.