Hi Friends
Robots.txt file contains some kind of text(not HTML)which you put on your site to tell to search engine which page you would like them to visit and which page you wouldn't like them to visit.
Robots.txt file has lots of importance as it allows spiders or crawlers to allow or disallow to crawl all pages of a website or a particular webpage.It provides you with more functionality than Meta robots tag which is available only partially to control behaviour of search engines.
Robots.txt in particular, ensures search engine bots to determine the pages that they should visit and the pages they should not. Some parts of the website might contain some sensitive data and we might not want the search engine bots to cache them and show it on the web publicly. Such pages can be disallowed by using robots.txt. Though robots.txt aren't a compulsion for every website.
The method on the site was invented later, to allow page authors who do not have access to the server configuration to control the indexing of their pages. It is less efficient than using robots.txt, because the HTML page must be downloaded to read the meta tags. Therefore, using robots.txt save bandwidth. In addition, the meta tag method only works for HTML pages and not images, scripts, CSS files, etc. Since the meta tag is an html tag, it can be interpreted, if displayed on an HTML page.
A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. file should have legal information and other secure information that owner would not want to sharing or search by search engine.
Robot .txt is very impotant file gogle page crowler
Good SEO Services recommend that robots.txt file must be formatted properly so all important files in your web site can be indexed
Sometimes you don't want Google bots to visit certain areas or pages of your website and get them indexed. Thats why you need Robot.txt
Robots.txt is simple txt file it is more important to getting good ranking in search engine. in that
Robots.txt file will give permission to the search engine spiders, which parts crawls or not.
Robots.txt file contains some kind of text(not HTML)which you put on your site to tell to search engine which page you would like them to visit and which page you wouldn't like them to visit.
Robots.txt is important to inform the search engine which pages or folders you want to crawl and which one you don't want to crawl. Also, you can mentioned which search engine you want to crawl your site and which search engine you don't want to crawl your site.
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored..
Robots.txt can assist you from avoiding the look for machines from crawling search engines, but like any other resources, Robots.txt can be not reliable at times so don't depend your top key information to a text file.
It allows Google bots and other robots to crawl into our sites index them ..
This is the main step in setting up an website.
Robot.txt file is very important for search engine. it is helpfull for search engines for knowing which pages should be indexed by them or which pages do not to be indexed by them.
Robots.txt file is useful If you want search engines to ignore any pages from your website on your website. It also helps search engine to easily get sitemap of your website.
You might be surprised to hear that one small text file, known as robots.txt, could be the downfall of your website. If you get the file wrong you could end up telling search engine robots not to crawl your site, meaning your web pages won't appear in the search results. Therefore, it's important that you understand the purpose of a robots.txt file and learn how to check you're using it correctly.
A robots.txt file gives instructions to web robots about the pages the website owner doesn't wish to be 'crawled'. For instance, if you didn't want your images to be listed by Google and other search engines, you'd block them using your robots.txt file.
It's depends on the situation or else both having their own importance.
Suppose you want to block indexing of only one page in Search Engine, than in this case robots meta tag will be a good option instead of robots.txt. Because if you are blocking a single page by using robots.txt then there is some chances to appear that URL in SERP, For Example check below FB page that is blocked by using robots.txt and its still appearing in SERP.
Suppose you want to block indexing of directory or folder then in this case robots.txt is best option. Because if we are planning to use robots tag then we have to add robots tag on all page and its time consuming process.
Crawling Budget:
One thing to notice is that the Google will still crawl the complete page even though we block any pages by using robots meta tag. But its vice-versa in case of robots.txt. So if you want to save Search Engine Crawl budget then robots.txt is good.
Quote from: chris.andrew on September 10, 2011, 07:18:30 AM
Hi Friends
Robots.txt file contains some kind of text(not HTML)which you put on your site to tell to search engine which page you would like them to visit and which page you wouldn't like them to visit.
A robots.txt file gives instructions to web robots about the pages the website owner doesn't wish to be 'crawled'.
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Robots.txt is a method that helps to instruct search engines bots, which pages to crawl & index on one website & which not to.
Bots are constantly crawling and indexing websites for relevant information to any given search query. This is the importance of robots.txt.
Robot .txt is very important file google page crawler