In order for search engines to rank your web site, they must index your pages and review your content. Instructing robots what to index and what to ignore results in only relevant content being indexed. The way search engines index a web site is by sending out a computer program called a 'spider'. By default, the spider will read everything on your site — even your system files. When the spider arrives at your site, it seeks for a file called 'robots.txt'. If found, the spider will follow the directions contained in this file. This file lists the folders that should not be indexed. The robots.txt file might look something like this: User-agent: * Disallow: /images/ Disallow: /cgi-bin/ Disallow: /photos/ As you can see, we have disallowed the spider to go into the above directories. This file sits in the root of your web site. You can disallow individual files as well. Disallow: email.htm.