Why robots.txt file for SEO

Started by anandiphone, January 18, 2012, 08:09:10 AM

anandiphone

This is very important text file that should be present on your website because using this file you can tell search engines not to crawl particular pages of your website.

helpout

The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web. The URL path (web address) of your robots.txt file should look like this...

Davidson

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:

http://www.yourwebsite.com/robots.txt

SmithRobert

Robots.txt file is really important. This file shows Search engine crawl any file from support directory as robots.txt file has instructions to search engine to not crawl any file from support directory. If you leave the Disallow line blank, you're telling the engine that all files may be indexed. In any case, you must enter a Disallow line for every User-agent record.

lillianabe

Robot txt file is also important factor. Robot.txt file gives instructions about the site to web robots. It gives instruction on which page to index and which not to search engine crawler.
Syntax of robot.txt file looks like

User-agent:*
Disallow:/

User agent*  means applies to all robots. The Disallow means the robot should not visit any pages in the site.

More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
      
website analysis spider tool for technical SEOA1 Website Analyzer
      
SEO tools for managing keywords and keyword listsA1 Keyword Research
      
complete website copier toolA1 Website Download
      
create custom website search enginesA1 Website Search Engine
      
scrape data into CSV, SQL and databasesA1 Website Scraper