Why we use Robots.txt File?

  • 19 Replies
  • 3704 Views
*

sneha1234

  • Newbie
  • *
  • 31
  • +0/-0
    • View Profile
Why we use Robots.txt File?
« on: March 09, 2012, 03:35:21 AM »
Robots.txt file is used when you have duplicate content that you don't want to be indexed, spammers use them to scan for email addresse. It is placed in the main directory.

*

AllenSantiago

  • Full Member
  • ***
  • 162
  • +0/-0
    • View Profile
Re: Why we use Robots.txt File?
« Reply #1 on: March 09, 2012, 05:42:55 AM »
Robots.txt is mainly useful to inform search engine crawlers regarding your website so that search engine can easily or properly indexing your web pages.

*

M. Rebecca

  • Full Member
  • ***
  • 199
  • +2/-0
    • View Profile
Re: Why we use Robots.txt File?
« Reply #2 on: March 09, 2012, 10:43:10 PM »
It serves another very important purpose that is automatic discovery. You can specify your Sitemap in your robot.txt file to enable better crawling by spiders.
300% more Adsense earnings with 12+ Keyword Research Tools

*

tochandresh

  • Newbie
  • *
  • 27
  • +0/-0
    • View Profile
    • eCommerce Portal Development
Re: Why we use Robots.txt File?
« Reply #3 on: March 14, 2012, 01:45:25 PM »

Robots.txt concept is only for Google search engine because only Google consider nofollow or disallow concept and Yahoo! Bind atavist don't follow these concept.

Robots.txt file is only for reducing spam and disallow automatically generated web pages in website.

*

tarunjangra

  • Jr. Member
  • **
  • 98
  • +0/-0
    • View Profile
    • Elgg Developers | Plugins | Elgg open Source | Programmers
Re: Why we use Robots.txt File?
« Reply #4 on: April 16, 2012, 06:23:32 AM »
robot.txt file is helpfull for reducing spaming.

*

Nichole Green

  • Full Member
  • ***
  • 151
  • +0/-0
    • View Profile
Re: Why we use Robots.txt File?
« Reply #5 on: April 17, 2012, 08:20:36 AM »
Robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed. Robots.txt file is to improve site indexation by telling search engine crawler to only index your content pages and to ignore other pages (i.e. monthly archives, categories folders or your admin files) that you do not want them to appear on the search index.

*

ericfox

  • Full Member
  • ***
  • 106
  • +0/-0
    • View Profile
Re: Why we use Robots.txt File?
« Reply #6 on: July 24, 2012, 02:04:34 AM »
Robots.txt allows you to specify which pages should not be crawled. The most useful and popular way of hiding the files from search engines is using the Robots.txt file. Actually robots meta tag is good too but some engines cannot read meta tags.


*

maxsamuel

  • Jr. Member
  • **
  • 91
  • +0/-0
    • View Profile
Re: Why we use Robots.txt File?
« Reply #7 on: August 20, 2012, 02:42:41 AM »
Robots.txt File is used for search engine to allow them that which page they have to crawl and which page they don't have to crawl. it is group of instruction  for search engine that how to crawl the whole site.

*

siyajoshi

  • Sr. Member
  • ****
  • 269
  • +0/-0
    • View Profile
Re: Why we use Robots.txt File?
« Reply #8 on: August 21, 2012, 03:51:04 AM »
Hello...
Robots.txt is a file that is used to exclude content from the crawling process of search engine spiders / bots. Robots.txt is also called the Robots Exclusion Protocol.In general, we prefer that our webpages are indexed by the search engines. But there may be some content that we don’t want to be crawled & indexed. Search engine like Google will not go to read the link that you have disallowed but some small search engine read that part that you have disallowed to see what is that so they waste their time.So we should avoid disallow field generally...
Thnx....:)

*

icecube media

  • Full Member
  • ***
  • 116
  • +0/-0
    • View Profile
Re: Why we use Robots.txt File?
« Reply #9 on: December 28, 2012, 10:29:57 AM »
Robots.txt file specifies Google which file to crawl and which file to not crawl.

*

petersmith5544

  • Newbie
  • *
  • 48
  • +0/-0
    • View Profile
    • Ascra Technologies
Re: Why we use Robots.txt File?
« Reply #10 on: January 04, 2013, 04:05:51 AM »
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:
http://www.yourwebsite.com/robots.txt

*

bedrik

  • Full Member
  • ***
  • 161
  • +0/-0
    • View Profile
Re: Why we use Robots.txt File?
« Reply #11 on: January 29, 2013, 06:37:05 AM »
Robots.txt always help Google crawlers or spider to know how to index the site. It provides instructions to spiders.

*

eface

  • Newbie
  • *
  • 8
  • +0/-0
    • View Profile
Re: Why we use Robots.txt File?
« Reply #12 on: February 13, 2013, 02:29:07 PM »
 Robot.txt  file is a simple text file, We are using this text file for security purpose. It blogs the search engines for certain page.

*

RachaelHetkins

  • Newbie
  • *
  • 47
  • +0/-0
    • View Profile
Re: Why we use Robots.txt File?
« Reply #13 on: March 16, 2013, 07:29:06 AM »
If you have a new website, you can utilize robots.txt file to tell search engines that how robots or crawl will index your website. Think that you have a website related to dog, and you have put a content related to cat on your website. Search engine assumes that this site does not focus on particularly on one theme, this thought is called theme bleeding, and search engine will do lower your page rank. To prevent theme bleeding in your site, you can utilize robot.txt file to stop search engine to index pages, which doesn't match with the theme of your site.

*

Alex Thompson

  • Hero Member
  • *****
  • 741
  • +0/-0
    • View Profile
    • Professional Web Design and Web Development
Re: Why we use Robots.txt File?
« Reply #14 on: April 05, 2013, 05:54:36 AM »
Robot.txt is way to instruct the search engine crawlers that what to crawl and what to not because there are so many file which useless foe search engines and we don't want to let them indexed.

*

Seo gurgaon

  • Full Member
  • ***
  • 139
  • +0/-0
  • SEO Services
    • View Profile
    • SEO Services Gurgaon
Re: Why we use Robots.txt File?
« Reply #15 on: May 01, 2013, 07:21:37 AM »
Robots.txt files are used to give instructions to the search engine crawlers about a domain or web page that the web page is to crawl, index or not not.

*

seoservicesezone

  • Jr. Member
  • **
  • 57
  • +0/-0
    • View Profile
    • IT Support Services
Re: Why we use Robots.txt File?
« Reply #16 on: May 03, 2013, 06:23:10 AM »
Robots.txt file is a text file in which we specify to search engines that which pages should not be crawled.

*

americostech

  • Newbie
  • *
  • 3
  • +0/-0
    • View Profile
    • Web Design
Re: Why we use Robots.txt File?
« Reply #17 on: May 28, 2013, 02:44:59 AM »
Where you can invite the crawler to your WebPages for  indexed by the search engines. But there may be some content that we don’t want to be crawled & indexed. Like the personal images folder, website administration folder, customer’s test folder of a web developer, no search value folders like cgi-bin, and many more. The main idea is we don’t want them to be indexed.



*

roger4321

  • Full Member
  • ***
  • 163
  • +0/-0
    • View Profile
Re: Why we use Robots.txt File?
« Reply #18 on: June 18, 2013, 11:13:32 AM »
just to instruct the search engine which page should be crawled and which not to be.

*

raveenasen

  • Jr. Member
  • **
  • 57
  • +0/-0
    • View Profile
Re: Why we use Robots.txt File?
« Reply #19 on: June 25, 2013, 03:41:26 AM »
Robots.txt is a text file that tells search engine spiders (what are search engine spiders?), also known as search engine robots, which parts of your website they can enter and which parts they can’t.The main reason why robots.txt would be used is to keep sensitive information private.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper