A1 Sitemap Generator Looping ...

Started by drgeorgep, May 01, 2016, 07:42:13 PM

drgeorgep

Hi ... for some reason my 7.2.0 version keeps looping. Just now, it has run for more than six hours and found more than 54 thousand URLs, although my site has fewer than 5000 URLs. This has been going on for a couple of weeks and is getting worse. I thought a fix might have come out, but, alas, not. Can someone offer helpful ideas? Thanks. dgp

Webhelpforums

#1
In 99% of such cases, it is because the website generates neverending duplicate/error urls.

From:
http://www.microsystools.com/products/sitemap-generator/help/creating-sitemaps-large-websites/

Quote
Check if your website is generating an infinite amount of unique URLs. If it does, it will cause the crawler to never stop as new unique page URLs are found all the time. A good method to discover and solve these kinds of problems is by:

  • Start a website scan.
  • Stop the website scan after e.g. half an hour.
  • Inspect if everything appears correct, i.e. if most of the URLs found seem correct.
       
Example #1
A website returns 200 instead of 404 for broken page URLs. Example of infinite pattern:
Original 1/broken.html links to 1/1/broken.html links to 1/1/1/broken.html etc.

Example #2
The website platform CMS generates a huge number of 100% duplicate URLs for each actual existing URL.




By the way, here's how to stop and resume website scans:
http://www.microsystools.com/products/sitemap-generator/help/sitemap-generator-resume-scan/
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

drgeorgep

Hi ... thanks for the prompt and clear response. I'll give it a try, right now, and get back. Thanks, again. dgp

More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
      
website analysis spider tool for technical SEOA1 Website Analyzer
      
SEO tools for managing keywords and keyword listsA1 Keyword Research
      
complete website copier toolA1 Website Download
      
create custom website search enginesA1 Website Search Engine
      
scrape data into CSV, SQL and databasesA1 Website Scraper