2. Webmaster Forums - Website and SEO Help
  3. Search Engines and Marketing
  4. Website Crawling and Indexing
  5. Is there a limit to robots.txt file?
Pages: [1]

Is there a limit to robots.txt file?

  • 3 Replies
  • 338 Views
*

bangalorewebguru

  • Website Design & Development Company in Bangalore
  • Sr. Member
  • ****
  • 288
  • +0/-0
  • Making Smartwork
    • View Profile
    • Web Designing Company in Bangalore,India
Is there a limit to robots.txt file?
« on: August 07, 2017, 02:59:56 AM »
Is there a limit to robots.txt file?
Logged
Web Designing Company Bangalore | Web Designing Companies Bangalore | Website Development Company Bangalore
*

Krina

  • Jr. Member
  • **
  • 90
  • +0/-0
    • View Profile
Re: Is there a limit to robots.txt file?
« Reply #1 on: August 13, 2018, 05:54:55 AM »
Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.
Logged
*

mtgfanboy

  • Full Member
  • ***
  • 114
  • +0/-0
    • View Profile
Re: Is there a limit to robots.txt file?
« Reply #2 on: August 24, 2018, 12:38:19 AM »
Yes, there is a limit of robots.txt file. Google only read 500 KB of robots.txt. Content which is after the maximum file size may be ignored.
Logged
*

nilomartseo

  • Jr. Member
  • **
  • 89
  • +0/-0
    • View Profile
Re: Is there a limit to robots.txt file?
« Reply #3 on: August 24, 2018, 05:38:16 AM »
Google has a limit of only being able to process up to 500KB of your robots.txt file. This is an important point, if you have a super heavy robots.txt file.
Logged
Pages: [1]
 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper