Use of robot.txt ?

  • 4 Replies
  • 225 Views
*

bangalorewebguru

  • Web Design Company Bangalore Web Guru
  • Sr. Member
  • ****
  • 288
  • +0/-0
  • Making Smartwork
    • View Profile
    • Web Design Company
Use of robot.txt ?
« on: June 08, 2017, 04:17:02 AM »
Use of robot.txt ?

*

RH-Calvin

  • Hero Member
  • *****
  • 729
  • +0/-0
    • View Profile
    • Reprise Hosting
Re: Use of robot.txt ?
« Reply #1 on: June 12, 2017, 09:51:47 AM »
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Cheap VPS | $1 VPS Hosting
Cheap Dedicated Hosting | Free Setup with IPMI

*

sirishasiri

  • Full Member
  • ***
  • 115
  • +0/-0
    • View Profile
Re: Use of robot.txt ?
« Reply #2 on: December 06, 2017, 06:56:37 AM »
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard
used by websites to communicate with web crawlers and other web robots. The standard specifies how to
inform the web robot about which areas of the website should not be processed or scanned.

*

compressjpg

  • Newbie
  • *
  • 5
  • +0/-0
    • View Profile
Re: Use of robot.txt ?
« Reply #3 on: December 14, 2017, 09:44:29 AM »
Thanks for sharing useful information...

*

hotmailserviceuk

  • Hotmail Technical Support Number
  • Newbie
  • *
  • 17
  • +0/-0
    • View Profile
    • Hotmail Contact Number
Re: Use of robot.txt ?
« Reply #4 on: December 20, 2017, 02:03:03 AM »
Hello Guys
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper