Is there a limit to robots.txt file?

  • 3 Replies
  • 338 Views
*

bangalorewebguru

  • Website Design & Development Company in Bangalore
  • Sr. Member
  • ****
  • 288
  • +0/-0
  • Making Smartwork
    • View Profile
    • Web Designing Company in Bangalore,India
Is there a limit to robots.txt file?
« on: August 07, 2017, 02:59:56 AM »
Is there a limit to robots.txt file?
Web Designing Company Bangalore | Web Designing Companies Bangalore | Website Development Company Bangalore

*

Krina

  • Jr. Member
  • **
  • 90
  • +0/-0
    • View Profile
Re: Is there a limit to robots.txt file?
« Reply #1 on: August 13, 2018, 05:54:55 AM »
Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.

*

mtgfanboy

  • Full Member
  • ***
  • 114
  • +0/-0
    • View Profile
Re: Is there a limit to robots.txt file?
« Reply #2 on: August 24, 2018, 12:38:19 AM »
Yes, there is a limit of robots.txt file. Google only read 500 KB of robots.txt. Content which is after the maximum file size may be ignored.

*

nilomartseo

  • Jr. Member
  • **
  • 89
  • +0/-0
    • View Profile
Re: Is there a limit to robots.txt file?
« Reply #3 on: August 24, 2018, 05:38:16 AM »
Google has a limit of only being able to process up to 500KB of your robots.txt file. This is an important point, if you have a super heavy robots.txt file.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper