Website Scan need too long time?

  • 1 Replies
  • 1107 Views
*

emperorkangxi

  • Newbie
  • *
  • 1
  • +0/-0
    • View Profile
Website Scan need too long time?
« on: December 03, 2014, 11:28:32 PM »
I start to use A1 sitemap generator.
and scan my website last 20 hours..is it normal?
I do not change any option, is it need to optimize to short the scanning time?

Please help me!!


*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1364
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
Re: Website Scan need too long time?
« Reply #1 on: December 04, 2014, 06:17:19 AM »
I took a look at your website (which you emailed) paused the scan after 30 minutes to inspect the resuls.

It looks like there might be a good opportunity to exclude vast ranges of URLs from website analysis and sitemap output!

From the screenshot I sent, you can see your website generates many similar URLs (e.g. searches for different keywords, sort orders etc.) It may even be your website inserts infinite or near-infinite amount of unique keywords into search pages? In that case - the scan will never end unless you exclude some URLs.

You know your own website best, but if you can exclude ranges / patterns of unwanted URLs, you can speed up the scan a lot.

You should exclude:
  • URLs with duplicate/similar content
  • URLs with very little content

While A1 Sitemap Generrator obeys robots.txt, nofollow, noindex, canonical etc. it can for large websites that generate *many* similar URLs be an advanage to exclude their URL pattterns in A1SG already during the website scan.

About excluding URLs see:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/

You can also see this help page that has general help on scanning large and/or slow websites:
http://www.microsystools.com/products/sitemap-generator/help/creating-sitemaps-large-websites/

About noindex, nofollow, canonical, robots.txt see this:
http://www.microsystools.com/products/sitemap-generator/help/sitemap-robots-noindex-nofollow/
MicrosysTools.com | Website and SEO Software for webmasters | A1 Sitemap Generator, A1 Website Analyzer etc.