Robots.txt file is used when you have duplicate content that you don't want to be indexed, spammers use them to scan for email addresse. It is placed in the main directory.

Robots.txt is mainly useful to inform search engine crawlers regarding your website so that search engine can easily or properly indexing your web pages.

It serves another very important purpose that is automatic discovery. You can specify your Sitemap in your robot.txt file to enable better crawling by spiders.

Robots.txt concept is only for Google search engine because only Google consider nofollow or disallow concept and Yahoo! Bind atavist don't follow these concept.

Robots.txt file is only for reducing spam and disallow automatically generated web pages in website.

robot.txt file is helpfull for reducing spaming.

Robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed. Robots.txt file is to improve site indexation by telling search engine crawler to only index your content pages and to ignore other pages (i.e. monthly archives, categories folders or your admin files) that you do not want them to appear on the search index.

Robots.txt allows you to specify which pages should not be crawled. The most useful and popular way of hiding the files from search engines is using the Robots.txt file. Actually robots meta tag is good too but some engines cannot read meta tags.

Robots.txt File is used for search engine to allow them that which page they have to crawl and which page they don't have to crawl. it is group of instruction  for search engine that how to crawl the whole site.

Hello...
Robots.txt is a file that is used to exclude content from the crawling process of search engine spiders / bots. Robots.txt is also called the Robots Exclusion Protocol.In general, we prefer that our webpages are indexed by the search engines. But there may be some content that we don't want to be crawled & indexed. Search engine like Google will not go to read the link that you have disallowed but some small search engine read that part that you have disallowed to see what is that so they waste their time.So we should avoid disallow field generally...
Thnx....

Robots.txt file specifies Google which file to crawl and which file to not crawl.

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:
http://www.yourwebsite.com/robots.txt

Robots.txt always help Google crawlers or spider to know how to index the site. It provides instructions to spiders.

 Robot.txt  file is a simple text file, We are using this text file for security purpose. It blogs the search engines for certain page.

If you have a new website, you can utilize robots.txt file to tell search engines that how robots or crawl will index your website. Think that you have a website related to dog, and you have put a content related to cat on your website. Search engine assumes that this site does not focus on particularly on one theme, this thought is called theme bleeding, and search engine will do lower your page rank. To prevent theme bleeding in your site, you can utilize robot.txt file to stop search engine to index pages, which doesn't match with the theme of your site.

Robot.txt is way to instruct the search engine crawlers that what to crawl and what to not because there are so many file which useless foe search engines and we don't want to let them indexed.

Robots.txt files are used to give instructions to the search engine crawlers about a domain or web page that the web page is to crawl, index or not not.

Robots.txt file is a text file in which we specify to search engines that which pages should not be crawled.

Where you can invite the crawler to your WebPages for  indexed by the search engines. But there may be some content that we don't want to be crawled & indexed. Like the personal images folder, website administration folder, customer's test folder of a web developer, no search value folders like cgi-bin, and many more. The main idea is we don't want them to be indexed.

just to instruct the search engine which page should be crawled and which not to be.

Robots.txt is a text file that tells search engine spiders (what are search engine spiders?), also known as search engine robots, which parts of your website they can enter and which parts they can't.The main reason why robots.txt would be used is to keep sensitive information private.