What is the limit of a robot.txt file?

  • 3 Replies
  • 125 Views
*

pihu147741

  • Jr. Member
  • **
  • 87
  • +0/-0
    • View Profile
    • Best Astrology Solution
What is the limit of a robot.txt file?
« on: May 08, 2018, 03:27:14 AM »
Hello friends,

What is the limit of a robot.txt file?

*

sinelogixtech

  • Hero Member
  • *****
  • 637
  • +0/-0
    • View Profile
Re: What is the limit of a robot.txt file?
« Reply #1 on: May 08, 2018, 06:25:11 AM »
Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out.

*

RH-Calvin

  • Hero Member
  • *****
  • 1080
  • +0/-0
    • View Profile
    • Reprise Hosting
Re: What is the limit of a robot.txt file?
« Reply #2 on: May 11, 2018, 01:04:17 AM »
Google actually crawls 500 KB of text in robots.txt file.
Cheap VPS | $1 VPS Hosting
Cheap Dedicated Servers | Free Setup with IPMI

*

Raj

  • Jr. Member
  • **
  • 89
  • +0/-0
    • View Profile
    • USAC
Re: What is the limit of a robot.txt file?
« Reply #3 on: July 27, 2018, 03:36:23 AM »
Hello friends,

What is the limit of a robot.txt file?

500 kb

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper