ACS Athens: American Community Schools

How do Search Engines Work?

Search engines do not really search the World Wide Web directly. Each one searches a database of web pages that it has harvested and cached. When you use a search engine, you are always searching a somewhat stale copy of the real web page. When you click on links provided in a search engine's search results, you retrieve the current version of the page.

Search engine databases are selected and built by computer robot programs called spiders. These "crawl" the web, finding pages for potential inclusion by following the links in the pages they already have in their database. They cannot use imagination or enter terms in search boxes that they find on the web.

If a web page is never linked from any other page, search engine spiders cannot find it. The only way a brand new page can get into a search engine is for other pages to link to it, or for a human to submit its URL for inclusion. All major search engines offer ways to do this.

After spiders find pages, they pass them on to another computer program for "indexing." This program identifies the text, links, and other content in the page and stores it in the search engine database's files so that the database can be searched by keyword and whatever more advanced approaches are offered, and the page will be found if your search matches its content. 

What Are "Meta-Search" Engines? How Do They Work?

In a meta-search engine, you submit keywords in its search box, and it transmits your search simultaneously to several individual search engines and their databases of web pages. Within a few seconds, you get back results from all the search engines queried. Meta-search engines do not own a database of Web pages; they send your search terms to the databases maintained by search engine companies.

Are "Smarter" Meta-Searchers Still Smarter?

"Smarter" meta-searcher technology includes clustering and linguistic analysis that attempts to show you themes within results, and some fancy textual analysis and display that can help you dig deeply into a set of results. However, neither of these technologies is any better than the quality of the search engine databases they obtain results from. This is the topic of an insightful article titled, "Some Cautionary Notes on Vivisimo," by librarian and professional researcher, Rita Vine of Working Faster. But here is another viewpoint favoring meta-searching by saying "More heads better than one."

Few meta-searchers allow you to delve into the largest, most useful search engine databases. They tend to return results from smaller and/or free search engines and miscellaneous free directories, often small and highly commercial.

Although we respect the potential of textual analysis and clustering technologies, we recommend directly searching individual search engines to get the most precise results, and using meta-searchers if you want to explore more broadly.

The meta-search tools listed here are "use at your own risk." We are not endorsing or recommending them.

Better Meta-Searchers

Meta-Search Engines for SERIOUS Deep Digging

How to Find Subject-Focused Directories for a Specific Topic, Discipline, or Field

There are thousands of specialized directories on practically every subject. If you want an overview, or if you feel you've searched long enough, try to find one. Often they are done by experts -- self-proclaimed or heavily credentialed. Here are some ways to find them:

Use any of the Subject Directories above to find more specific directories. Here are some tips:

• In ipl2 or Infomine, look for your subject as you would for any other purpose, and keep your eyes open for sites that look like directories. Read through the descriptions. Sometimes these resources are identified as "Directories, "Virtual Libraries," or "Gateway Pages."
• In Yahoo! and Google directories, try adding the terms web directories to your subject keyword term:  

EXAMPLES:

civil war  web directories

weddings  web directories

• In About.com, search by topic and look for pages that are described as "101" or "guides" or a "directory." About.com is written by "Guides" who, themselves, often are experts in the sections they manage. Sometimes they write excellent overviews of a topic. 

Directory of Search Engines and Subject-specific Search Engines 

General Search Engines:

HotVsNot.Com - [New Window] - Internet web directory organized by subject.

About - [New Window] - Directory providing highly targeted topics overseen by subject specialists dedicated to helping you get what you want quickly and easily.

AltaVista - [New Window] - AltaVista indexes the full text of Web pages, so you can find the most obscure information easily by typing specific terms: the names of movies, books, songs, companies. alta vista

Ask Jeeves! - [New Window] - Search the web in plain English and receive a concise list of answers with their exact location.

Bing - [New Window] - Microsoft's search engine.

Excite - [New Window] - General portal and search engine.

Fast - [New Window] - Advanced search functionality allows users to quickly pinpoint precisely the information they seek.

Galaxy - [New Window] - A searchable Internet directory providing contextually relevant information by integrating state-of-the-art technology with the human touch.

Google - [New Window] - A Search engine that provides query results based on the number and quality of links to a web site in addition to search terms.

HotBot - [New Window] - HotBot provides a streamlined yet powerful starting point for searching the Internet. FAST, Inktomi, and Teoma, results are provided.

Lycos - [New Window] - A leading web portal combining Web search and navigation resources, deep content in vertical areas, communications and personalization tools, and a complete shopping center.

Northern Light Search - [New Window] - Search engine which enables you to search the Web and information from premium sources. Web results are combined with information from premium material in one search, giving you access to books, magazines, databases, and newswires.

SearchHippo - [New Window] - Provides a crisp and clean web search experience. Results from spidered sites are ordered based on a variety of factors including link-to frequency, clicked-on frequency, and URL directory depth.

SurfWax

SurfWax's patent-pending design is the first to make searching a "visual process," seamlessly integrating meaning-based search with key knowledge-finding elements for effective association and recall.

Switchboard - [New Window] - Internet directory. Yellowpages, whitepages, and phonebook. Look up businesses online, look up people, find addresses and phone numbers. Also provides zip codes, maps and directions.

Yahoo! - [New Window] - A searchable directory of Internet web sites organized in a vast collection of categories and sub-categories.

Zeal - [New Window] - User contributed and maintained directory featuring website ratings and reviews.

A9.com - [New Window] - A9.com offers users search results from five powerful information sources, which are presented through convenient selectable and adjustable columns: Web and image search provided by Google, book text of more than 100,000 titles from Amazon.com's Search Inside The Book, movie information from the Internet Movie Database, and reference information (encyclopedia, dictionary, etc.) through GuruNet.com. Additionally, A9.com is a search engine with a memory as it returns results from the user's information, so with every search, users will see results from their own history, bookmarks, and diary.

Active Search Results - [New Window] - Active Search Results (ASR) is an independent Internet Search Engine using a proprietary page ranking technology with Millions of popular Web sites indexed. ASR has its own spiders visiting Web sites daily that are submitted to ASR and crawlers that index other popular Web sites on the Internet. Active Search Results does not depend on nor utilize any other search engines or directories for its search results. All search results are contained within Active Search Results' indexes and databases.

AOL Search

AtoZ.com - [New Window] - Categorized links of web sites.

Better Brain - [New Window] - Search 11 major search engines at once, search MP3 music, newsgroups or browse the open directory.

Cannylink - [New Window] - Web directory.

Clickey - [New Window] - The Internet's first domain name only search engine, allowing you to find web sites by keywords, country, state and city.

Cuil - [New Window] - Rather than rely on superficial popularity metrics, Cuil searches for and ranks pages based on their content and relevance. It finds a page with your keywords, and stays on that page and analyze the rest of its content, its concepts, their inter-relationships and the page's coherency. Then offers you helpful choices and suggestions until you find the page you want and that you know is out there.