Started by drgeorgep, May 01, 2016, 07:42:13 PM
QuoteCheck if your website is generating an infinite amount of unique URLs. If it does, it will cause the crawler to never stop as new unique page URLs are found all the time. A good method to discover and solve these kinds of problems is by:Start a website scan.Stop the website scan after e.g. half an hour.Inspect if everything appears correct, i.e. if most of the URLs found seem correct. Example #1A website returns 200 instead of 404 for broken page URLs. Example of infinite pattern:Original 1/broken.html links to 1/1/broken.html links to 1/1/1/broken.html etc.Example #2The website platform CMS generates a huge number of 100% duplicate URLs for each actual existing URL.