Website Scan need too long time?

  • 1 Replies
  • 1248 Views
*

emperorkangxi

  • Newbie
  • *
  • 1
  • +0/-0
    • View Profile
Website Scan need too long time?
« on: December 03, 2014, 11:28:32 PM »
I start to use A1 sitemap generator.
and scan my website last 20 hours..is it normal?
I do not change any option, is it need to optimize to short the scanning time?

Please help me!!


*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1379
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
Re: Website Scan need too long time?
« Reply #1 on: December 04, 2014, 06:17:19 AM »
I took a look at your website (which you emailed) paused the scan after 30 minutes to inspect the resuls.

It looks like there might be a good opportunity to exclude vast ranges of URLs from website analysis and sitemap output!

From the screenshot I sent, you can see your website generates many similar URLs (e.g. searches for different keywords, sort orders etc.) It may even be your website inserts infinite or near-infinite amount of unique keywords into search pages? In that case - the scan will never end unless you exclude some URLs.

You know your own website best, but if you can exclude ranges / patterns of unwanted URLs, you can speed up the scan a lot.

You should exclude:
  • URLs with duplicate/similar content
  • URLs with very little content

While A1 Sitemap Generrator obeys robots.txt, nofollow, noindex, canonical etc. it can for large websites that generate *many* similar URLs be an advanage to exclude their URL pattterns in A1SG already during the website scan.

About excluding URLs see:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/

You can also see this help page that has general help on scanning large and/or slow websites:
http://www.microsystools.com/products/sitemap-generator/help/creating-sitemaps-large-websites/

About noindex, nofollow, canonical, robots.txt see this:
http://www.microsystools.com/products/sitemap-generator/help/sitemap-robots-noindex-nofollow/
MicrosysTools.com | Website and SEO Software for webmasters | A1 Sitemap Generator, A1 Website Analyzer etc.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper