Search Engines 101

At a workshop for Learning Centre leads, I was part of the audience for a teleconference with Rita Vine. Ms. Vine is an authority on web searching and gave us a great overview of search engines. A few points she made:

There are only 6 search engines (defined as a tool that crawls the web and downloads web pages (a spider) into a searchable database and also enables keyword searching of that database). These are Google, Yahoo!, Ask, MSN, Gigablast and Exalead. Only Exalead is not based in the US.

Results:
She talked about the priority of results - what comes first in the search results. This variety but the primary method for Google and Yahoo! is link popularity - the number of links that other web pages have made to a page makes it more popular and rank higher in the results. However I didn't know that .gov and .edu websites get extra weighting. .gov is always US and .edu is most often US.

Ms. Vine talked about advertising and she said that Google is the world's largest advertising agency! In US-owned engines, ads in search results must be marked. Google does this most clearly.

Which one to use?
She talked about which search engine to use. There are three factors when comparing search engines - the rank ordering, the size of the database and the features. In her opinion, relevance ranking outweighs database size: if the first few results are relevant, it doesn't matter whether or not the database has billions or zillions of sites in their database! However you are better off searching a larger database, providing that it does produce relevant results. Google has few features but good relevance.

Search strategies:
I liked her suggestions for search strategies.
"Don't think subject - think keyword pattern matching". She reminded us that keywords must match exactly.

Two tips that I will keep in mind - she suggested that we omit needless words when searching because putting in needless words can restrict results. Her example - searching for "repetitive strain injury" (without quotation marks, by the way) would not match "repetitive strain injuries" or "repetitive strain syndrome".

A related suggestion is to use quotation marks only for proper names. Even asking for phrases like "water quality" rather than the keywords water quality limit your results.

Rita's rule of five - if you get more than 5 commercial / .com links OR 5 ads, you need to use another search engine as well for that search.

She talked about when search engines don't work:
* when the information is not likely to be available for free - e.g. market research reports. (Also, of course, is the material that we have in All In One Search, like the directories and the new automobile repair manual resource).

* when someone can make money from the information

* anything controversial - see actually says "remotely controversial". Ms. Vine mentioned the topic of medical use of marijuana - there are more con pages than pro.

* complex questions that don't have easy answers - like our reference questions sometimes! Questions that need interpretation of results or need meshing of a couple of sources.

* Canadian information. The databases only search for .ca domain names and the repetition of Canada or Canadian (or other geographical names). She pointed out that her website searchingfaster.com which is hosted in the US and has a .com domain name wouldn't be retrieved with the Canada filter.

For Canadian content, she suggested the Virtual Reference Library (yeah!)

* the "invisible web" - sites that are invisible to spiders (software that builds the search engine database), blocked or licensed sites, and some databases whose design doesn't lend itself to the search engine software. For example the Canadian Patent Database is not searched. Also Pubmed, the huge US government health database.

Rita's search tip: find the database then search it. Use any search engine and type the keywords with the word database: e.g. +database patents

Things keep changing with searching. And just when we get comfy with all this, Web 2.0 (social bookmarking / tagging) comes along!