Sitemap Generator Runs for Hours on one site

  • 1 Replies
  • 1917 Views
*

aplusnetsolutions

  • Newbie
  • *
  • 2
  • +0/-0
    • View Profile
Sitemap Generator Runs for Hours on one site
« on: April 05, 2011, 10:03:05 AM »
Hello,
I use the sitemap generator for 30+ sites.  For most it runs fine.  But there are a few for which it runs and runs with no end in sight.  For this site, paesanos1604.com, it ran for 3+ hours and was still running.  It finds database records and indexes those.  But when I looked at it there were over 21,000 Internal sitemap URLs.  I know there were not 21,000 database records for that site.

Any idea what to do on a situation like this?
Thanks,
Greg

*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1415
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
Re: Sitemap Generator Runs for Hours on one site
« Reply #1 on: April 05, 2011, 10:58:41 AM »
Disable "Easy mode":
http://www.microsystools.com/products/sitemap-generator/help/easy-sitemap-generator-mode/

Configure for using resume:
http://www.microsystools.com/products/sitemap-generator/help/sitemap-generator-resume-scan/

I would disable:
Scan website | Crawler options | Apply "webmaster" and "list" filters after website scan.
(This means A1SG will show you URLs it would normally delete when website scan stops. It could be URLs filtered off by robots.txt or a multitude other reasons. However, the sitemap.xml won't contain these URLs.)

Run a website scan an hour or whatever. See the URLs found. Are there URLs you believe should not be analyzed for links or be included in sitemap output?

Add Them to analyze exclude filters:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/

Add them to output exclude filters:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper