Robots.txt file – Do I Need It?

A robots.txt file basically tells search engines which part of your site not to crawl. The format is to put it in your top level directory in the format. It is useful in restricting parts of your site you do not want indexed by certain search engines. Expamples include:

 

User-agent: *
Disallow:
Examples of Other formats include the following

Disallow All crawlers

User-agent: *
Disallow: /

Restrict folder

User-agent: *
Disallow: /private/

Restrict file to all robots

User-agent: *
Disallow: /directory/file.html

Some may wonder whether it is worth having a robots.txt file at all if you want to give search engine crawlers unrestricted access. Matt Cutts has answered in his weekly digest that it is useful to have the files even if it is to say disallow none.

Enhanced by Zemanta