What is robots.txt file and use?

  • 11 Replies
  • 2167 Views
*

anandiphone

  • Full Member
  • ***
  • 199
  • +0/-0
    • View Profile
What is robots.txt file and use?
« on: September 02, 2011, 08:39:39 AM »
Kindly tell me the actual use of robots.txt file and its use?

*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1383
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
Re: What is robots.txt file and use?
« Reply #1 on: September 03, 2011, 04:30:01 PM »
To limit search engines and other crawlers from crawling URLs you do not want. (You can also e.g. control crawl-delay to prevent nice-behaving crawler robots from overloading your website server)

You can also use it to point to an XML sitemap
MicrosysTools.com | Website and SEO Software for webmasters | A1 Sitemap Generator, A1 Website Analyzer etc.

*

evelyn8411

  • Newbie
  • *
  • 14
  • +0/-0
    • View Profile
Re: What is robots.txt file and use?
« Reply #2 on: September 04, 2011, 07:50:48 PM »
Robot text is used to direct search engines about where they can crawl your site. Of course, only approved SE follow this. Vicious crawlers will not follow your robot text and crawl everything it wants/

*

John23

  • Jr. Member
  • **
  • 93
  • +0/-0
    • View Profile
Re: What is robots.txt file and use?
« Reply #3 on: September 05, 2011, 01:33:28 AM »
In point of view Robots txt files are use to give permission to search engine weather to index the webpaeg or foolow the page.there are several instruction for it.

*

alvypetersen

  • Newbie
  • *
  • 17
  • +0/-0
    • View Profile
Re: What is robots.txt file and use?
« Reply #4 on: September 05, 2011, 02:39:53 AM »
Robots.txt is a text file that tells search engine spiders, also known as search engine robots, which parts of your website they can enter and which parts they can’t.

*

dpfocanil

  • Newbie
  • *
  • 13
  • +0/-0
    • View Profile
    • Ireland Hotels
Re: What is robots.txt file and use?
« Reply #5 on: September 06, 2011, 05:22:04 AM »
A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. It divided into sections by the robot crawler's User Agent name.

User-agent: * Disallow:

*

hannahraasch

  • Guest
Re: What is robots.txt file and use?
« Reply #6 on: September 06, 2011, 06:51:27 AM »
Simply, the robots.txt is a very simple text file that is placed on our root directory. For example www.example.com/robots.txt. This file tells search engine and other robots which areas of our site they are allowed to visit and index.
« Last Edit: September 30, 2011, 10:02:18 AM by Webhelpforums »

*

ykgforum

  • Full Member
  • ***
  • 104
  • +0/-0
    • View Profile
    • Staff Uniform
Re: What is robots.txt file and use?
« Reply #7 on: October 18, 2011, 07:34:20 AM »
For google page crawler

*

Nichole Green

  • Full Member
  • ***
  • 151
  • +0/-0
    • View Profile
Re: What is robots.txt file and use?
« Reply #8 on: October 25, 2011, 06:00:06 AM »
The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Most websites have files and folders that are not relevant for search engines (like images or admin files) therefore creating a robots.txt file can actually improve your website indexation.
A robots.txt is a simple text file that can be created with Notepad. If you are using WordPress a sample robots.txt file would be:
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
If you are not using WordPress just substitute the Disallow lines with files or folders on your website that should not be crawled, for instance:
User-agent: *
Disallow: /images/
Disallow: /cgi-bin/
Disallow: /any other folder to be excluded/

*

Black Ninja

  • Die Hard
  • Full Member
  • ***
  • 204
  • +0/-0
  • “Don't cry because it's over,smile bcoz it happene
    • View Profile
    • Great Journal
Re: What is robots.txt file and use?
« Reply #9 on: November 02, 2011, 02:20:24 AM »
Robots.txt allows you to specify which pages should not be crawled. Pages that don't get crawled can still rank for keywords and show up in search results.

*

infoanil

  • Newbie
  • *
  • 12
  • +0/-0
    • View Profile
    • Franchise Litigation
Re: What is robots.txt file and use?
« Reply #10 on: November 02, 2011, 03:08:01 AM »
Robots.txt is a file that is used to leave out content from the crawling process of search engine spiders / bots. WebPages are indexed by the search engines. But there may be some content that we don’t want to be crawled & indexed. The main plan is we don’t want them to be indexed.

*

LucieJason

  • Jr. Member
  • **
  • 52
  • +0/-0
    • View Profile
Re: What is robots.txt file and use?
« Reply #11 on: November 28, 2011, 02:02:15 AM »
Robots.txt is veryuseful and helpful file. When a crawler crawl a website, it read robots.txt initially. Robots.txt is used widely to instruct the crawler that which page has to be crawled and which is not.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper