Robots, Crawlers, Worms, and Spiders

"Crawl around and find what we can"

Continuously running program (robot, crawler, worm, spider) pursuing hyper-links throughout Internet

Start with set of documents. Identify new places to explore by looking at outbound links. Visit those links. Index most useful terms.

Lycos project at Carnegie-Mellon

"Lycos" comes from arachnid family Lycosidae -- large ground spiders, very speedy and active at night, catching their prey by pursuit rather than in a web

Lycos has database of over 50 million Web pages

Receives about 4 million visits a day