Can't Get A1 Sitemap Generator to crawl a certain directory

Started by rudypohl, January 12, 2012, 03:36:00 PM

rudypohl

Hi There:

I can't Get A1 Sitemap Generator to crawl a certain directory. I've tried everything. Set the chMod to 777 makes no difference

Webhelpforums

#1
If you try to crawl the directory directly, please make sure the directory does not redirect to e.g. the domain root.

If above or similar is the case, and you have pages in that directory linked from elsewhere, you can use a combination of:

1)
Click off "Easy mode"
http://www.microsystools.com/products/sitemap-generator/help/easy-sitemap-generator-mode/

2)
Configure "start search paths"
http://www.microsystools.com/products/sitemap-generator/help/root-aliases-start-paths/

3)
Configure include/exclude paths to analyze:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/

4)
Configure include/exclude paths to output and keep:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/

...

You may also want to check if it is a robots.txt, noindex, nofollow or similar issue:
http://www.microsystools.com/products/sitemap-generator/help/sitemap-robots-noindex-nofollow/
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
      
website analysis spider tool for technical SEOA1 Website Analyzer
      
SEO tools for managing keywords and keyword listsA1 Keyword Research
      
complete website copier toolA1 Website Download
      
create custom website search enginesA1 Website Search Engine
      
scrape data into CSV, SQL and databasesA1 Website Scraper