Can't Get A1 Sitemap Generator to crawl a certain directory

  • 1 Replies
  • 2143 Views
*

rudypohl

  • Newbie
  • *
  • 1
  • +0/-0
    • View Profile
Can't Get A1 Sitemap Generator to crawl a certain directory
« on: January 12, 2012, 03:36:00 PM »
Hi There:

I can't Get A1 Sitemap Generator to crawl a certain directory. I've tried everything. Set the chMod to 777 makes no difference

*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1413
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
Re: Can't Get A1 Sitemap Generator to crawl a certain directory
« Reply #1 on: January 12, 2012, 07:57:20 PM »
If you try to crawl the directory directly, please make sure the directory does not redirect to e.g. the domain root.

If above or similar is the case, and you have pages in that directory linked from elsewhere, you can use a combination of:

1)
Click off "Easy mode"
http://www.microsystools.com/products/sitemap-generator/help/easy-sitemap-generator-mode/

2)
Configure "start search paths"
http://www.microsystools.com/products/sitemap-generator/help/root-aliases-start-paths/

3)
Configure include/exclude paths to analyze:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/

4)
Configure include/exclude paths to output and keep:
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/

...

You may also want to check if it is a robots.txt, noindex, nofollow or similar issue:
http://www.microsystools.com/products/sitemap-generator/help/sitemap-robots-noindex-nofollow/
« Last Edit: January 12, 2012, 07:59:33 PM by Webhelpforums »
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper