Sitemap generator 3.x frozen on Windows Seven with 50% CPU load !

Hi there,

It seems that I cannot use anymore the new version of the software on my Windows Seven PC...
The 2.3.5 is still working well (only 2% or 3% of CPU usage and stopping well) but when I tried to use any 3.x versions (included the latest version 3.1.5), the software seems to be frozen in an endless loop using 25% to 50% of my CPU, forcing me to quit and the only way I have is to kill all the sitemap.exe process to stop it...

There must have been changes in the way 3.x versions are working compared to 2.x versions ?

Please help me solve this major issue for us...

Thanks in advance

Hi,

Can you email/pm website address? That would be very helpful in solving the problem

I have not really heard of this problem from anyone else. (A few people reporting *after crawl nearing maybe half million or one million URLs* that it can stall. But that has always been the case when using default settings. You can configure A1SG not to collect extended data which increases capacity. In 3.x there is a bit more data collected, but not much.)

My best guess would still be that you have scanned a very big website and it ran out of memory and maybe of that then also stalls CPU...  But it could be *anything*. I need you to PM/email:
* website address.
* the project file you are using.

Hi,

Thanks for your prompt answer. 

The site is not so big, less than 7000 URLS, its address is http://www.sibarthrealestate.com/sibarthrealestate

I do not store anything in the configuration of the Data Collection (not even the log) to prevent slow crawl and overload the server.
What is your email so that I can send your the .ini file ?

Best regards

Check contact or just email to: info _at_ microsystools.com

I am running a scan against your website now using 3.1.5 (all default settings)

With only 7000 URLs it's not a general capacity problem. But maybe there is something very specific project/website combination that exposes an issue I will look forward to receiving your project file

Meanwhile I will initiate a couple of tests with default settings

I have run 4 successful tests with Javascript enabled, error logging enabled etc. all sorts of combinations. No problems.


I am running a 5th project right now with <forms> handling enabled.
And I see major memory spiking there...

I am guess there's a huge list of URLs that build up somewhere to be listed/analyzed.
Imagine you have 3 options each of 100 choices (e.g. age or whatever). That gives 100*100*100 = 1 million URLs that get queued for test/lookup (and then listed and/or analyzed)

I have not received any email from you yet, but my guess is above is to blame. Quite possibly 2.x did not support <forms> related HTML tags as well as 3.x does meaning 2.x possibly did not catch as many unique combinations. This is my *best* guess sofar. But of course I can't 100% rule out a bug somewhere in the <forms> code. I will keep my eye on it

Maybe I will add a debug mode info, so one can see amount of URLs queued for lookup/test (when A1SG tests a queued URL it first check if the URL already has been analyzed/listed already. If it has, A1SG just updates links-to/linked-by counts and similar data)

Hi,

I had a quite busy week-end. Sorry for not keeping in touch.

Thank you very much for your tests.

I just sent an email with an attached .ini file of our project (on the "info" address) if it helps you understand the memory increase and see what's wrong...
I do not remember checking the forms in this project, but following all others links...

So I will wait for your feedback and thoughts on this.

Thanks again

Best regards

Thanks Under all circumstances, I will recommend you simply use default settings (File | New Project). It has been default for a long time to use GET instead of HEAD requests. And it appears to be much faster on your website as well. (Which is common which us why GET was made default.) Anyways, I am running your test project now

It seems that when using HEAD + ZLib there is (at some point) a disagreement between the ZLib decompression I use and what the server ZLib compression uses. I am looking into this issue. Hopefully I will have it solved soon to at least avoid this sort of showstopper problem! Thanks for reporting this issue

Under all circumstances (even when this issue is solved) I recommend you use default settings

I am still running a couple of large tests to make sure I caught everything for the 3.1.6 release. (If I didn't, remaining will be fixed for 3.1.7)

Hi,
Thanks for your time, I appreciate.
I have not tested yet the 3.1.6 update and will do it tomorrow for sure...
In the meantime, further to your advice I created a new project without modifying nothing from the original presets and it worked well, like a charm !
I do not remember exactly what were the specific options I modified in the initial project, but anyway... it seems to work well now with your original default configuration. I will keep using it !

I hope this issue could help resolve hidden bugs and improve your really great software.

Thanks for your efficient support.