Importance of robot.txt

  • 17 Replies
  • 2564 Views
*

chris.andrew

  • Newbie
  • *
  • 6
  • +0/-0
    • View Profile
    • Wireless Medical Alert
Importance of robot.txt
« on: September 10, 2011, 07:18:30 AM »
Hi Friends

Robots.txt file contains some kind of text(not HTML)which you put on your site to tell to search engine which page you would like them to visit and which page you wouldn't like them to visit.

*

charlott

  • Newbie
  • *
  • 5
  • +0/-0
    • View Profile
Re: Importance of robot.txt
« Reply #1 on: September 13, 2011, 07:16:35 AM »
Robots.txt file has lots of importance as it allows spiders or crawlers to allow or disallow to crawl all pages of a website or a particular webpage.It provides you with more functionality than Meta robots tag which is available only partially to control behaviour of search engines.






*

TTOM

  • Newbie
  • *
  • 26
  • +0/-0
    • View Profile
Re: Importance of robot.txt
« Reply #2 on: September 26, 2011, 02:32:42 AM »
Robots.txt in particular, ensures search engine bots to determine the pages that they should visit and the pages they should not. Some parts of the website might contain some sensitive data and we might not want the search engine bots to cache them and show it on the web publicly. Such pages can be disallowed by using robots.txt. Though robots.txt aren't a compulsion for every website.

*

markwaugh12

  • Newbie
  • *
  • 9
  • +0/-0
    • View Profile
Re: Importance of robot.txt
« Reply #3 on: September 27, 2011, 04:36:49 AM »
The method on the site was invented later, to allow page authors who do not have access to the server configuration to control the indexing of their pages. It is less efficient than using robots.txt, because the HTML page must be downloaded to read the meta tags. Therefore, using robots.txt save bandwidth. In addition, the meta tag method only works for HTML pages and not images, scripts, CSS files, etc. Since the meta tag is an html tag, it can be interpreted, if displayed on an HTML page.

please see forum signature rules :)

*

infoanil

  • Newbie
  • *
  • 12
  • +0/-0
    • View Profile
    • Franchise Litigation
Re: Importance of robot.txt
« Reply #4 on: October 07, 2011, 08:19:43 AM »
A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. file should have legal information and other secure information that owner would not want to sharing or search by search engine.

*

ykgforum

  • Full Member
  • ***
  • 104
  • +0/-0
    • View Profile
    • Staff Uniform
Re: Importance of robot.txt
« Reply #5 on: October 18, 2011, 07:31:38 AM »
Robot .txt is very impotant file gogle page crowler

*

AshleyVilligant

  • Newbie
  • *
  • 6
  • +0/-0
    • View Profile
Re: Importance of robot.txt
« Reply #6 on: November 22, 2011, 07:12:31 AM »
Good SEO Services recommend that robots.txt file must be formatted properly so all important files in your web site can be indexed
http://www.pherx.com/

*

KGM1

  • Newbie
  • *
  • 12
  • +0/-0
    • View Profile
Re: Importance of robot.txt
« Reply #7 on: December 21, 2011, 04:48:02 PM »
Sometimes you don't want Google bots to visit certain areas or pages of your website and get them indexed. Thats why you need Robot.txt

*

lillianabe

  • Full Member
  • ***
  • 182
  • +0/-0
    • View Profile
Re: Importance of robot.txt
« Reply #8 on: December 26, 2011, 08:07:26 AM »
Robots.txt is simple txt file it is more important to getting good ranking in search engine. in that
Robots.txt file will give permission to the  search engine spiders, which parts crawls or not.

*

seniorlivingca

  • Jr. Member
  • **
  • 60
  • +0/-0
    • View Profile
Re: Importance of robot.txt
« Reply #9 on: January 19, 2012, 12:24:42 AM »
Robots.txt file contains some kind of text(not HTML)which you put on your site to tell to search engine which page you would like them to visit and which page you wouldn't like them to visit.

*

kelwin

  • Newbie
  • *
  • 22
  • +0/-0
    • View Profile
Re: Importance of robot.txt
« Reply #10 on: January 23, 2012, 11:54:35 PM »
Robots.txt is important to inform the search engine which pages or folders you want to crawl and which one you don't want to crawl. Also, you can mentioned which search engine you want to crawl your site and which search engine you don't want to crawl your site.

*

Davidson

  • Newbie
  • *
  • 10
  • +0/-0
    • View Profile
Re: Importance of robot.txt
« Reply #11 on: January 25, 2012, 07:22:34 AM »
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored..

*

farisboorem

  • Newbie
  • *
  • 5
  • +0/-0
    • View Profile
    • mobilerepeatershop
Re: Importance of robot.txt
« Reply #12 on: March 17, 2012, 07:39:41 AM »
Robots.txt can assist you from avoiding the look for machines from crawling search engines, but like any other resources, Robots.txt can be not reliable at times so don't depend your top key information to a text file.

*

Daniel

  • Newbie
  • *
  • 13
  • +0/-0
    • View Profile
Re: Importance of robot.txt
« Reply #13 on: March 19, 2012, 09:59:23 AM »
It allows Google bots and other robots to crawl into our sites index them ..
This is the main step in setting up an website.

*

tarunjangra

  • Jr. Member
  • **
  • 98
  • +0/-0
    • View Profile
    • Elgg Developers | Plugins | Elgg open Source | Programmers
Re: Importance of robot.txt
« Reply #14 on: April 10, 2012, 06:05:55 AM »
Robot.txt file is very important for search engine. it is helpfull for search engines for knowing which pages should be indexed by them or which pages do not to be indexed by them.

*

darshan.nimblechapps

  • Newbie
  • *
  • 7
  • +0/-0
    • View Profile
    • Nimblechapps
Re: Importance of robot.txt
« Reply #15 on: September 28, 2016, 05:34:47 AM »
Robots.txt file is useful If you want search engines to ignore any pages from your website on your website. It also helps search engine to easily get sitemap of your website.

*

Williams Reus

  • Newbie
  • *
  • 45
  • +0/-0
    • View Profile
Re: Importance of robot.txt
« Reply #16 on: September 28, 2016, 07:14:39 PM »
You might be surprised to hear that one small text file, known as robots.txt, could be the downfall of your website.  If you get the file wrong you could end up telling search engine robots not to crawl your site, meaning your web pages won’t appear in the search results.  Therefore, it’s important that you understand the purpose of a robots.txt file and learn how to check you’re using it correctly.

A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’.  For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.

*

benben1

  • Newbie
  • *
  • 30
  • +0/-0
    • View Profile
Re: Importance of robot.txt
« Reply #17 on: October 03, 2016, 12:11:53 PM »
It’s depends on the situation or else both having their own importance.

Suppose you want to block indexing of only one page in Search Engine, than in this case robots meta tag will be a good option instead of robots.txt. Because if you are blocking a single page by using robots.txt then there is some chances to appear that URL in SERP, For Example check below FB page that is blocked by using robots.txt and its still appearing in SERP.
Suppose you want to block indexing of directory or folder then in this case robots.txt is best option. Because if we are planning to use robots tag then we have to add robots tag on all page and its time consuming process.

Crawling Budget:
One thing to notice is that the Google will still crawl the complete page even though we block any pages by using robots meta tag. But its vice-versa in case of robots.txt. So if you want to save Search Engine Crawl budget then robots.txt is good.
The Video Marketers Profit Mining SystemVid Reaper Review|Good Software find video[/URL

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper