Image Sitemap taking forever!

Started by Cody J., February 07, 2013, 05:11:57 PM

Cody J.

Hello,

I run an e-commerce site that has quite a few images and am trying to setup a image sitemap. To exclude lots of the temporary redirects etc our commerce puts into place I limited the output to url response codes of 0 and 200 which came out OK on my previous sitemaps - just to skip any 301's, 302's, 404's (hope they dont exist), etc.

Thing is... it's taking hours! I let it run overnight the other day and it took 7 hrs 30 minutes and then when I went to build the sitemap, it errored and said I need to pick 'google image sitemap' from the drop-down.. in which I did before I scanned and it still error'd. So today I am re-running it with the response codes in place.

Currently at 3 hrs 23 minute.
http://imgur.com/S3fyrD6 - EXAMPLE PIC

Is this normal? Thanks!


Webhelpforums

#1
Doing the 301/404 exclusions in "scan website | output filters" won't always save you time because:

1)
before A1SG knows an URL errors (404 etc.) it has already downloaded it. (You can override this before and configure A1SG to use HEAD(ers) requests before GET. However, some servers refuse / are buggy when it comes to HEAD requests.)

2)
even if an URL is excluded in "output filters" it may still be an URL that passes "analysis filters", and if so, it is still downloaded and analyzed for links (just not included in output if excluded in "output filters")

...

Concerning the missing images. Are you sure you are no hosting images on external domains? And if you do, did you use the image preset where images are hosted on external domains? If you used the external preset, that will also explain why scan takes long time since lots of external URLs are then checked.

Anyhow, you can just bumb connections + threads in "scan website | crawler engine" the - default speed configuration in A1SG is about 5-10% of max. (Depends on many factors, so hard to give an exact number)

...

See also:
http://www.microsystools.com/products/sitemap-generator/help/image-xml-sitemaps/

If you still have problems, drop an email to support with link to the website:
http://www.microsystools.com/home/contact.php
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
      
website analysis spider tool for technical SEOA1 Website Analyzer
      
SEO tools for managing keywords and keyword listsA1 Keyword Research
      
complete website copier toolA1 Website Download
      
create custom website search enginesA1 Website Search Engine
      
scrape data into CSV, SQL and databasesA1 Website Scraper