See Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
      
website analysis spider tool for technical SEOA1 Website Analyzer
      
SEO tools for managing keywords and keyword listsA1 Keyword Research
      
complete website copier toolA1 Website Download
      
create custom website search enginesA1 Website Search Engine
      
scrape data into CSV, SQL and databasesA1 Website Scraper

A1 Sitemap Generator Looping ...

  • 2 Replies
  • 4270 Views
*

drgeorgep

  • Newbie
  • *
  • 5
  • +0/-0
    • View Profile
A1 Sitemap Generator Looping ...
« on: May 01, 2016, 07:42:13 PM »
Hi ... for some reason my 7.2.0 version keeps looping. Just now, it has run for more than six hours and found more than 54 thousand URLs, although my site has fewer than 5000 URLs. This has been going on for a couple of weeks and is getting worse. I thought a fix might have come out, but, alas, not. Can someone offer helpful ideas? Thanks. dgp

*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1440
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
Re: A1 Sitemap Generator Looping ...
« Reply #1 on: May 02, 2016, 06:29:29 AM »
In 99% of such cases, it is because the website generates neverending duplicate/error urls.

From:
http://www.microsystools.com/products/sitemap-generator/help/creating-sitemaps-large-websites/

Quote
Check if your website is generating an infinite amount of unique URLs. If it does, it will cause the crawler to never stop as new unique page URLs are found all the time. A good method to discover and solve these kinds of problems is by:
  • Start a website scan.
  • Stop the website scan after e.g. half an hour.
  • Inspect if everything appears correct, i.e. if most of the URLs found seem correct.
       
Example #1
A website returns 200 instead of 404 for broken page URLs. Example of infinite pattern:
Original 1/broken.html links to 1/1/broken.html links to 1/1/1/broken.html etc.

Example #2
The website platform CMS generates a huge number of 100% duplicate URLs for each actual existing URL.



By the way, here's how to stop and resume website scans:
http://www.microsystools.com/products/sitemap-generator/help/sitemap-generator-resume-scan/
« Last Edit: May 02, 2016, 06:31:10 AM by Webhelpforums »
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

*

drgeorgep

  • Newbie
  • *
  • 5
  • +0/-0
    • View Profile
Re: A1 Sitemap Generator Looping ...
« Reply #2 on: May 02, 2016, 12:58:51 PM »
Hi ... thanks for the prompt and clear response. I'll give it a try, right now, and get back. Thanks, again. dgp

 

Learn More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
      
website analysis spider tool for technical SEOA1 Website Analyzer
      
SEO tools for managing keywords and keyword listsA1 Keyword Research
      
complete website copier toolA1 Website Download
      
create custom website search enginesA1 Website Search Engine
      
scrape data into CSV, SQL and databasesA1 Website Scraper