Why robots.txt file for SEO

  • 4 Replies
  • 1150 Views
*

anandiphone

  • Full Member
  • ***
  • 199
  • +0/-0
    • View Profile
Why robots.txt file for SEO
« on: January 18, 2012, 08:09:10 AM »
This is very important text file that should be present on your website because using this file you can tell search engines not to crawl particular pages of your website.

*

helpout

  • Newbie
  • *
  • 15
  • +0/-0
    • View Profile
Re: Why robots.txt file for SEO
« Reply #1 on: January 28, 2012, 03:21:35 AM »
The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web. The URL path (web address) of your robots.txt file should look like this...

*

Davidson

  • Newbie
  • *
  • 10
  • +0/-0
    • View Profile
Re: Why robots.txt file for SEO
« Reply #2 on: February 01, 2012, 04:08:33 AM »
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:

http://www.yourwebsite.com/robots.txt

*

SmithRobert

  • Newbie
  • *
  • 16
  • +0/-0
    • View Profile
    • Centrifugal Pump
Re: Why robots.txt file for SEO
« Reply #3 on: February 02, 2012, 01:42:17 AM »
Robots.txt file is really important. This file shows Search engine crawl any file from support directory as robots.txt file has instructions to search engine to not crawl any file from support directory. If you leave the Disallow line blank, you're telling the engine that all files may be indexed. In any case, you must enter a Disallow line for every User-agent record.

*

lillianabe

  • Full Member
  • ***
  • 182
  • +0/-0
    • View Profile
Re: Why robots.txt file for SEO
« Reply #4 on: February 02, 2012, 07:25:50 AM »
Robot txt file is also important factor. Robot.txt file gives instructions about the site to web robots. It gives instruction on which page to index and which not to search engine crawler.
Syntax of robot.txt file looks like

User-agent:*
Disallow:/

User agent*  means applies to all robots. The Disallow means the robot should not visit any pages in the site.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper