Website Scan need too long time?

Started by emperorkangxi, December 03, 2014, 11:28:32 PM

emperorkangxi

I start to use A1 sitemap generator.
and scan my website last 20 hours..is it normal?
I do not change any option, is it need to optimize to short the scanning time?

Please help me!!


Webhelpforums

I took a look at your website (which you emailed) paused the scan after 30 minutes to inspect the resuls.

It looks like there might be a good opportunity to exclude vast ranges of URLs from website analysis and sitemap output!

From the screenshot I sent, you can see your website generates many similar URLs (e.g. searches for different keywords, sort orders etc.) It may even be your website inserts infinite or near-infinite amount of unique keywords into search pages? In that case - the scan will never end unless you exclude some URLs.

You know your own website best, but if you can exclude ranges / patterns of unwanted URLs, you can speed up the scan a lot.

You should exclude:

  • URLs with duplicate/similar content
  • URLs with very little content

While A1 Sitemap Generrator obeys robots.txt, nofollow, noindex, canonical etc. it can for large websites that generate *many* similar URLs be an advanage to exclude their URL pattterns in A1SG already during the website scan.

About excluding URLs see:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/

You can also see this help page that has general help on scanning large and/or slow websites:
http://www.microsystools.com/products/sitemap-generator/help/creating-sitemaps-large-websites/

About noindex, nofollow, canonical, robots.txt see this:
http://www.microsystools.com/products/sitemap-generator/help/sitemap-robots-noindex-nofollow/
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
      
website analysis spider tool for technical SEOA1 Website Analyzer
      
SEO tools for managing keywords and keyword listsA1 Keyword Research
      
complete website copier toolA1 Website Download
      
create custom website search enginesA1 Website Search Engine
      
scrape data into CSV, SQL and databasesA1 Website Scraper