Webmaster Forums - Website and SEO Help

Microsys Products and Webmaster Tools => A1 Sitemap Generator => Topic started by: rudypohl on January 12, 2012, 03:36:00 PM

Title: Can't Get A1 Sitemap Generator to crawl a certain directory
Post by: rudypohl on January 12, 2012, 03:36:00 PM
Hi There:

I can't Get A1 Sitemap Generator to crawl a certain directory. I've tried everything. Set the chMod to 777 makes no difference
Title: Re: Can't Get A1 Sitemap Generator to crawl a certain directory
Post by: Webhelpforums on January 12, 2012, 07:57:20 PM
If you try to crawl the directory directly, please make sure the directory does not redirect to e.g. the domain root.

If above or similar is the case, and you have pages in that directory linked from elsewhere, you can use a combination of:

1)
Click off "Easy mode"
http://www.microsystools.com/products/sitemap-generator/help/easy-sitemap-generator-mode/ (http://www.microsystools.com/products/sitemap-generator/help/easy-sitemap-generator-mode/)

2)
Configure "start search paths"
http://www.microsystools.com/products/sitemap-generator/help/root-aliases-start-paths/ (http://www.microsystools.com/products/sitemap-generator/help/root-aliases-start-paths/)

3)
Configure include/exclude paths to analyze:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/ (http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/)

4)
Configure include/exclude paths to output and keep:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/ (http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/)

...

You may also want to check if it is a robots.txt, noindex, nofollow or similar issue:
http://www.microsystools.com/products/sitemap-generator/help/sitemap-robots-noindex-nofollow/ (http://www.microsystools.com/products/sitemap-generator/help/sitemap-robots-noindex-nofollow/)