What is robot.txt file?

If we want to block some data from search engine then we used this robot.txt file. This file must be uploaded on root server. This is the first file which is crawled by search engine first.

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.