What Is Robot.txt & Why We Use It?

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

Web Robot are known as crawlers, spiders. And robot.txt use to give instructions to these Web robots. Using this file a website owner can block robots to follow some link in website.

The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl.

Just answer me that if you really don't know about robots and its use, why you uploaded robots.txt file on your website? 

The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.

By listing webpages in Robot.txt we can say google to which webpage need to crow and which one not.

We use robot.txt file to restrict the search engine crawlers from crawling the page and one of the exaples is the log-in module in your site.

Quote from: asad1997 on August 23, 2019, 01:12:14 AM
Web Robot are known as crawlers, spiders. And robot.txt use to give instructions to these Web robots. Using this file a website owner can block robots to follow some link in website.

Yup.You have explained very well.we can block robots to follow some link in website with the help of robots.txt

A robots.txt file tells crawlers which search engine crawlers pages or files can or can not ask from your site. It is used primarily to avoid overloading your site with requests; it is not a mechanism to keep the web page of Google.