What is robots.txt?

  • 12 Replies
  • 246 Views
*

MoulanaRafi

  • Jr. Member
  • **
  • 87
  • +0/-0
    • View Profile
    • Husband Wife Problem
What is robots.txt?
« on: January 23, 2018, 12:40:31 AM »
What is robots.txt?

*

sophia009

  • Newbie
  • *
  • 33
  • +0/-0
    • View Profile
Re: What is robots.txt?
« Reply #1 on: January 23, 2018, 03:40:52 AM »
Hi

The robots.txt file is a text file to limit outstanding bots (robots, search engine crawlers) on a website from a website or some page on the website. Using a robots.txt file and with the denial direction, we can restrict booting or search engine crawling programs from websites or certain folders and files.

Hope it helps you.

*

vishwa01

  • Newbie
  • *
  • 13
  • +0/-0
    • View Profile
    • Tourist Guide
Re: What is robots.txt?
« Reply #2 on: January 24, 2018, 12:00:44 AM »
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

*

fayeseom

  • Full Member
  • ***
  • 230
  • +0/-0
    • View Profile
Re: What is robots.txt?
« Reply #3 on: January 24, 2018, 12:21:06 AM »
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen   sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

*

saketbhai

  • Full Member
  • ***
  • 132
  • +0/-0
    • View Profile
    • Vashikaran Specialist Baba
Re: What is robots.txt?
« Reply #4 on: January 24, 2018, 02:42:47 AM »
Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.

*

praveenitech1

  • Full Member
  • ***
  • 124
  • +0/-0
    • View Profile
Re: What is robots.txt?
« Reply #5 on: March 01, 2018, 04:24:28 AM »
The robots.txt file is a text file to limit outstanding bots (robots, search engine crawlers) on a website from a website or some page on the website.

*

vishnu priya

  • Visit My Sites
  • Full Member
  • ***
  • 122
  • +0/-0
    • View Profile
    • Salon Services at Home
Re: What is robots.txt?
« Reply #6 on: March 05, 2018, 01:55:28 AM »
Robots.txt is a text file created to instruct search engine robots, how to crawl pages on their website.

*

Xplor Bangalore

  • Newbie
  • *
  • 10
  • +0/-0
    • View Profile
Re: What is robots.txt?
« Reply #7 on: April 05, 2018, 04:02:17 AM »
Robots.txt is a page for Search engines information about the pages a company wants indexed or crawled. You can find this page by doing to YOURDOMAIN/robots.txt.

*

kiruthi_18

  • Full Member
  • ***
  • 114
  • +0/-0
    • View Profile
    • Sweet Shop
Re: What is robots.txt?
« Reply #8 on: April 06, 2018, 04:23:55 AM »
Robots.txt is a text file webmasters make to instruct search engine robot show to traffic at a snail's pace pages on their website.

*

RH-Calvin

  • Hero Member
  • *****
  • 1040
  • +0/-0
    • View Profile
    • Reprise Hosting
Re: What is robots.txt?
« Reply #9 on: April 09, 2018, 07:44:09 AM »
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Cheap VPS | $1 VPS Hosting
Cheap Dedicated Servers | Free Setup with IPMI

*

efusionworld

  • Newbie
  • *
  • 38
  • +0/-0
    • View Profile
Re: What is robots.txt?
« Reply #10 on: April 13, 2018, 12:49:21 AM »
Robots.txt is a text file located in the site's root directory that specifies for search engine's crawlers & spiders what website pages & files you want or don't want them to visit.

*

sandy mary

  • Full Member
  • ***
  • 109
  • +0/-0
  • Digital Marketing Analyst in Chennai
    • View Profile
    • IndigoWomensCenter
Re: What is robots.txt?
« Reply #11 on: April 20, 2018, 08:32:42 AM »
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots how to crawl and index pages on their website.

*

Davidsmith0143

  • Full Member
  • ***
  • 198
  • +0/-0
    • View Profile
Re: What is robots.txt?
« Reply #12 on: April 30, 2018, 02:33:15 AM »
Robot.txt is the text file created in the website by webmaster which hold the instruction for the search engine robots or crawlers.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper