Sitemap Generator Runs for Hours on one site

Started by aplusnetsolutions, April 05, 2011, 10:03:05 AM

aplusnetsolutions

Hello,
I use the sitemap generator for 30+ sites.  For most it runs fine.  But there are a few for which it runs and runs with no end in sight.  For this site, paesanos1604.com, it ran for 3+ hours and was still running.  It finds database records and indexes those.  But when I looked at it there were over 21,000 Internal sitemap URLs.  I know there were not 21,000 database records for that site.

Any idea what to do on a situation like this?
Thanks,
Greg

Webhelpforums

Disable "Easy mode":
http://www.microsystools.com/products/sitemap-generator/help/easy-sitemap-generator-mode/

Configure for using resume:
http://www.microsystools.com/products/sitemap-generator/help/sitemap-generator-resume-scan/

I would disable:
Scan website | Crawler options | Apply "webmaster" and "list" filters after website scan.
(This means A1SG will show you URLs it would normally delete when website scan stops. It could be URLs filtered off by robots.txt or a multitude other reasons. However, the sitemap.xml won't contain these URLs.)

Run a website scan an hour or whatever. See the URLs found. Are there URLs you believe should not be analyzed for links or be included in sitemap output?

Add Them to analyze exclude filters:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/

Add them to output exclude filters:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
      
website analysis spider tool for technical SEOA1 Website Analyzer
      
SEO tools for managing keywords and keyword listsA1 Keyword Research
      
complete website copier toolA1 Website Download
      
create custom website search enginesA1 Website Search Engine
      
scrape data into CSV, SQL and databasesA1 Website Scraper