What is Robots.txt file?

  • 3 Replies
  • 105 Views
*

MoulanaRafi

  • Jr. Member
  • **
  • 86
  • +0/-0
    • View Profile
    • Husband Wife Problem
What is Robots.txt file?
« on: June 08, 2018, 01:32:17 AM »
What is Robots.txt file?

*

earthkind

  • Newbie
  • *
  • 13
  • +0/-0
    • View Profile
Re: What is Robots.txt file?
« Reply #1 on: June 08, 2018, 02:13:56 AM »
1. Allow Full Access:

User-agent: *
Disallow:

2. Block All Access:

User-agent: *
Disallow: /

*

manisthajain

  • Newbie
  • *
  • 14
  • +0/-0
    • View Profile
    • Hair Transplant India
Re: What is Robots.txt file?
« Reply #2 on: June 11, 2018, 02:19:50 AM »
Search Engine first comes to this file and read given Instructions by site owner that who pages need to crawl and not.

*

RH-Calvin

  • Hero Member
  • *****
  • 841
  • +0/-0
    • View Profile
    • Reprise Hosting
Re: What is Robots.txt file?
« Reply #3 on: June 13, 2018, 07:52:31 AM »
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Cheap VPS | $1 VPS Hosting
Cheap Dedicated Hosting | Free Setup with IPMI

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper