Hi ... for some reason my 7.2.0 version keeps looping. Just now, it has run for more than six hours and found more than 54 thousand URLs, although my site has fewer than 5000 URLs. This has been going on for a couple of weeks and is getting worse. I thought a fix might have come out, but, alas, not. Can someone offer helpful ideas? Thanks. dgp
In 99% of such cases, it is because the website generates neverending duplicate/error urls.
From:
http://www.microsystools.com/products/sitemap-generator/help/creating-sitemaps-large-websites/ (http://www.microsystools.com/products/sitemap-generator/help/creating-sitemaps-large-websites/)
Quote
Check if your website is generating an infinite amount of unique URLs. If it does, it will cause the crawler to never stop as new unique page URLs are found all the time. A good method to discover and solve these kinds of problems is by:
- Start a website scan.
- Stop the website scan after e.g. half an hour.
- Inspect if everything appears correct, i.e. if most of the URLs found seem correct.
Example #1
A website returns 200 instead of 404 for broken page URLs. Example of infinite pattern:
Original 1/broken.html links to 1/1/broken.html links to 1/1/1/broken.html etc.
Example #2
The website platform CMS generates a huge number of 100% duplicate URLs for each actual existing URL.
By the way, here's how to stop and resume website scans:
http://www.microsystools.com/products/sitemap-generator/help/sitemap-generator-resume-scan/ (http://www.microsystools.com/products/sitemap-generator/help/sitemap-generator-resume-scan/)
Hi ... thanks for the prompt and clear response. I'll give it a try, right now, and get back. Thanks, again. dgp