Website Scan need too long time?

Started by emperorkangxi, December 03, 2014, 11:28:32 PM

emperorkangxi

I start to use A1 sitemap generator.
and scan my website last 20 hours..is it normal?
I do not change any option, is it need to optimize to short the scanning time?

Please help me!!


Webhelpforums

I took a look at your website (which you emailed) paused the scan after 30 minutes to inspect the resuls.

It looks like there might be a good opportunity to exclude vast ranges of URLs from website analysis and sitemap output!

From the screenshot I sent, you can see your website generates many similar URLs (e.g. searches for different keywords, sort orders etc.) It may even be your website inserts infinite or near-infinite amount of unique keywords into search pages? In that case - the scan will never end unless you exclude some URLs.

You know your own website best, but if you can exclude ranges / patterns of unwanted URLs, you can speed up the scan a lot.

You should exclude:

  • URLs with duplicate/similar content
  • URLs with very little content

While A1 Sitemap Generrator obeys robots.txt, nofollow, noindex, canonical etc. it can for large websites that generate *many* similar URLs be an advanage to exclude their URL pattterns in A1SG already during the website scan.

About excluding URLs see:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/

You can also see this help page that has general help on scanning large and/or slow websites:
http://www.microsystools.com/products/sitemap-generator/help/creating-sitemaps-large-websites/

About noindex, nofollow, canonical, robots.txt see this:
http://www.microsystools.com/products/sitemap-generator/help/sitemap-robots-noindex-nofollow/
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

More About Our Webmaster Tools for Windows and Mac