Image Sitemap taking forever!

Started by Cody J., February 07, 2013, 05:11:57 PM

Cody J.


I run an e-commerce site that has quite a few images and am trying to setup a image sitemap. To exclude lots of the temporary redirects etc our commerce puts into place I limited the output to url response codes of 0 and 200 which came out OK on my previous sitemaps - just to skip any 301's, 302's, 404's (hope they dont exist), etc.

Thing is... it's taking hours! I let it run overnight the other day and it took 7 hrs 30 minutes and then when I went to build the sitemap, it errored and said I need to pick 'google image sitemap' from the drop-down.. in which I did before I scanned and it still error'd. So today I am re-running it with the response codes in place.

Currently at 3 hrs 23 minute. - EXAMPLE PIC

Is this normal? Thanks!


Doing the 301/404 exclusions in "scan website | output filters" won't always save you time because:

before A1SG knows an URL errors (404 etc.) it has already downloaded it. (You can override this before and configure A1SG to use HEAD(ers) requests before GET. However, some servers refuse / are buggy when it comes to HEAD requests.)

even if an URL is excluded in "output filters" it may still be an URL that passes "analysis filters", and if so, it is still downloaded and analyzed for links (just not included in output if excluded in "output filters")


Concerning the missing images. Are you sure you are no hosting images on external domains? And if you do, did you use the image preset where images are hosted on external domains? If you used the external preset, that will also explain why scan takes long time since lots of external URLs are then checked.

Anyhow, you can just bumb connections + threads in "scan website | crawler engine" the - default speed configuration in A1SG is about 5-10% of max. (Depends on many factors, so hard to give an exact number)


See also:

If you still have problems, drop an email to support with link to the website:
TechSEO360 |  | A1 Sitemap Generator, A1 Website Analyzer etc.

More About Our Webmaster Tools for Windows and Mac