A crawler (also called a spider) is a program that searches the Internet and locates new public resources. These resources can include web documents and other types of files. Crawlers report their finds to an Internet database (called indexing).
Each search engine has its own crawler program that crawls the Internet for new material to list on the search engine. Spider technology is necessary because the amount of information being added to the Internet on a daily basis is more than any human team can index.