Author Topic: Sitemap Generator Runs for Hours on one site  (Read 445 times)

aplusnetsolutions

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
Sitemap Generator Runs for Hours on one site
« on: April 05, 2011, 08:03:05 AM »
Hello,
I use the sitemap generator for 30+ sites.  For most it runs fine.  But there are a few for which it runs and runs with no end in sight.  For this site, paesanos1604.com, it ran for 3+ hours and was still running.  It finds database records and indexes those.  But when I looked at it there were over 21,000 Internal sitemap URLs.  I know there were not 21,000 database records for that site.

Any idea what to do on a situation like this?
Thanks,
Greg

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • Posts: 1003
  • Karma: +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • Webmaster and Website Help Forums
Re: Sitemap Generator Runs for Hours on one site
« Reply #1 on: April 05, 2011, 08:58:41 AM »
Disable "Easy mode":
http://www.microsystools.com/products/sitemap-generator/help/easy-sitemap-generator-mode/

Configure for using resume:
http://www.microsystools.com/products/sitemap-generator/help/sitemap-generator-resume-scan/

I would disable:
Scan website | Crawler options | Apply "webmaster" and "list" filters after website scan.
(This means A1SG will show you URLs it would normally delete when website scan stops. It could be URLs filtered off by robots.txt or a multitude other reasons. However, the sitemap.xml won't contain these URLs.)

Run a website scan an hour or whatever. See the URLs found. Are there URLs you believe should not be analyzed for links or be included in sitemap output?

Add Them to analyze exclude filters:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/

Add them to output exclude filters:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/
MicrosysTools.com | Website and SEO Software for webmasters | A1 Sitemap Generator, A1 Website Analyzer etc.

 

Microsys

Twitter   Facebook   LinkedIn

Google+   YouTube   Pinterest
WebHelpForums

WHF at Twitter   WHF at Facebook   WHF at Google+
Join Our Newsletter
More information