What is Robots.txt file?

  • 6 Replies
  • 387 Views
*

MoulanaRafi

  • Jr. Member
  • **
  • 87
  • +0/-0
    • View Profile
    • Husband Wife Problem
What is Robots.txt file?
« on: June 08, 2018, 01:32:17 AM »
What is Robots.txt file?

*

earthkind

  • Newbie
  • *
  • 17
  • +0/-0
    • View Profile
Re: What is Robots.txt file?
« Reply #1 on: June 08, 2018, 02:13:56 AM »
1. Allow Full Access:

User-agent: *
Disallow:

2. Block All Access:

User-agent: *
Disallow: /

*

manisthajain

  • Newbie
  • *
  • 20
  • +0/-0
    • View Profile
Re: What is Robots.txt file?
« Reply #2 on: June 11, 2018, 02:19:50 AM »
Search Engine first comes to this file and read given Instructions by site owner that who pages need to crawl and not.

*

RH-Calvin

  • Hero Member
  • *****
  • 1182
  • +0/-0
    • View Profile
    • Reprise Hosting
Re: What is Robots.txt file?
« Reply #3 on: June 13, 2018, 07:52:31 AM »
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Cheap VPS | $1 VPS Hosting
Cheap Dedicated Servers | Free Setup with IPMI

*

Mercy Michal

  • Jr. Member
  • **
  • 53
  • +0/-0
    • View Profile
Re: What is Robots.txt file?
« Reply #4 on: July 19, 2018, 02:46:40 AM »
Robots.txt file is the robots exclusion protocol. It hleps indicates the lists of we bpages that are what pages are allowed and what pages are disallowed from search engine crawl

*

Christiana Robert

  • Jr. Member
  • **
  • 71
  • +0/-0
    • View Profile
Re: What is Robots.txt file?
« Reply #5 on: July 23, 2018, 04:51:43 AM »
Search engine first read Robot.txt file instruction. In this file, all instructions are given by website owner like what are the pages are allowed crawl by Google and what are the pages are disallowed crawl by Google

*

kiruthi_18

  • Full Member
  • ***
  • 115
  • +0/-0
    • View Profile
    • Sweet Shop
Re: What is Robots.txt file?
« Reply #6 on: July 23, 2018, 01:55:59 PM »
Website owner create the Robot.txt file for giving instruction to search engine. Inside the file the site owner gave instructions which URL's are allowed crawl by search engine and which URL's are disallowed crawl by search engine

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper