I used A1 site generator to create a site map of my website imcloud9.com and it worked perfectly. At the time the website had only folders that contained html files. It did not have an index.html (or variant). Yesterday I installed wordpress in the root directory of the website. Now the only results I get are based on the blog folders. The other folders which contain html files are not crawled anymore. I spent time playing with different settings but was not successful. I was able to make the crawl work by manually setting "Beyond website root path, initiate scanning from paths", but that can be tedious. Is there another way to make it automatically crawl the other folders in the root path?
Hi,
It is *probably* because A1SG does no see/find links to your old URLs.
Different solutions:
1)
If you have *any* kind of "hub", "index" or "main" page(s) that contains URLs to some of the missed URLs (which then again possibly link to other URLs etc.) you would only need to add those in the "start searches from". If you do not have that, then maybe add the folders? (Depending on server configuration, the webserver will list HTML files found in the folder.)
http://www.microsystools.com/products/sitemap-generator/help/root-aliases-start-paths/ (http://www.microsystools.com/products/sitemap-generator/help/root-aliases-start-paths/)
2)
You can import a list of URLs. Done much like export, just import instead :)
http://www.microsystools.com/products/sitemap-generator/help/website-data-export-xml-csv/ (http://www.microsystools.com/products/sitemap-generator/help/website-data-export-xml-csv/)
Then use the "Recrawl" functionality in A1SG
http://www.microsystools.com/products/sitemap-generator/help/sitemap-generator-resume-scan/ (http://www.microsystools.com/products/sitemap-generator/help/sitemap-generator-resume-scan/)
If you find neither of above is suitable, but you have an idea that would solve your problem, I am always happy to receive feedback and ideas! :)