Preventing crawling

To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt files in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots. When a search Engine visit a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed, and will instruct the robot as to which pages are not to be crawled.

As a search engine crawler keep a cached copy of the file, it on occasion crawl pages a webmaster does not wish crawled. Pages prevented from being crawled include login specific pages as shopping carts and user specific content as search results from internal searches.