Webmaster Forums - Website and SEO Help

Microsys Products and Webmaster Tools => A1 Sitemap Generator => Topic started by: aplusnetsolutions on April 05, 2011, 10:03:05 AM

Title: Sitemap Generator Runs for Hours on one site
Post by: aplusnetsolutions on April 05, 2011, 10:03:05 AM
I use the sitemap generator for 30+ sites.  For most it runs fine.  But there are a few for which it runs and runs with no end in sight.  For this site, paesanos1604.com, it ran for 3+ hours and was still running.  It finds database records and indexes those.  But when I looked at it there were over 21,000 Internal sitemap URLs.  I know there were not 21,000 database records for that site.

Any idea what to do on a situation like this?
Title: Re: Sitemap Generator Runs for Hours on one site
Post by: Webhelpforums on April 05, 2011, 10:58:41 AM
Disable "Easy mode":

Configure for using resume:

I would disable:
Scan website | Crawler options | Apply "webmaster" and "list" filters after website scan.
(This means A1SG will show you URLs it would normally delete when website scan stops. It could be URLs filtered off by robots.txt or a multitude other reasons. However, the sitemap.xml won't contain these URLs.)

Run a website scan an hour or whatever. See the URLs found. Are there URLs you believe should not be analyzed for links or be included in sitemap output?

Add Them to analyze exclude filters:

Add them to output exclude filters: