Where you can invite the crawler to your WebPages for indexed by the search engines. But there may be some content that we don't want to be crawled & indexed. Like the personal images folder, website administration folder, customer's test folder of a web developer, no search value folders like cgi-bin, and many more. The main idea is we don't want them to be indexed.