what are the uses of robots.txt

  • 6 Replies
  • 276 Views
*

Jessicad

  • pgdme business college in chennai
  • Full Member
  • ***
  • 215
  • +0/-0
  • AIMED
    • View Profile
    • AIMED Academy for foreign languages
what are the uses of robots.txt
« on: December 18, 2018, 08:33:41 AM »
what are the uses of robots.txt

*

Netparticle

  • Sr. Member
  • ****
  • 288
  • +0/-0
    • View Profile
    • Netparticle
Re: what are the uses of robots.txt
« Reply #1 on: December 18, 2018, 11:16:11 PM »
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

*

rachana

  • Newbie
  • *
  • 25
  • +0/-0
    • View Profile
    • Mobile apps development companies India
Re: what are the uses of robots.txt
« Reply #2 on: December 18, 2018, 11:46:19 PM »
Robots.txt can be used for following purposes:
1. Robots.txt file indicates whether the pages of their website should be crawled or not.
2. To hide few pages that you don't want search engines to show in the SERP.
3. It shows the search engines where the XML sitemap is located so that they can find new pages quickly.

*

sinelogixtech

  • Hero Member
  • *****
  • 833
  • +0/-0
    • View Profile
Re: what are the uses of robots.txt
« Reply #3 on: December 18, 2018, 11:59:31 PM »
Robots.txt is useful for preventing the indexation of the parts of any online content that website owners do not want to display.

*

sinelogixtech

  • Hero Member
  • *****
  • 833
  • +0/-0
    • View Profile
Re: what are the uses of robots.txt
« Reply #4 on: December 19, 2018, 11:35:51 PM »
Using this file a website owner can block robots to follow some link in website. Ex: User-agent: * Disallow: / Above code means website owner restricting all (“*”) to visit whole website(“/”).

*

RH-Calvin

  • Hero Member
  • *****
  • 1314
  • +0/-0
    • View Profile
    • Reprise Hosting
Re: what are the uses of robots.txt
« Reply #5 on: December 20, 2018, 12:50:05 AM »
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Cheap VPS | $1 VPS Hosting
Cheap Dedicated Servers | Free Setup with IPMI

*

andrewtiwsan

  • Newbie
  • *
  • 5
  • +0/-0
    • View Profile
Re: what are the uses of robots.txt
« Reply #6 on: January 23, 2019, 05:37:34 AM »
thx for this intresting topic. nice information.
You can discover more here. Here you can find a lot of useful information.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper