This is very important text file that should be present on your website because using this file you can tell search engines not to crawl particular pages of your website.
The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web. The URL path (web address) of your robots.txt file should look like this...
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.
The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:
http://www.yourwebsite.com/robots.txt
Robots.txt file is really important. This file shows Search engine crawl any file from support directory as robots.txt file has instructions to search engine to not crawl any file from support directory. If you leave the Disallow line blank, you're telling the engine that all files may be indexed. In any case, you must enter a Disallow line for every User-agent record.
Robot txt file is also important factor. Robot.txt file gives instructions about the site to web robots. It gives instruction on which page to index and which not to search engine crawler.
Syntax of robot.txt file looks like
User-agent:*
Disallow:/
User agent* means applies to all robots. The Disallow means the robot should not visit any pages in the site.