A1 causes 403 error with disabled directory browsing

Started by Troffel, December 30, 2010, 08:16:31 AM

Troffel

I use .htaccess to disable directory browsing on my sites (Options -Indexes), i.e. when you use the address http://www.mysite.com/directory1 and their is no index.html file, you get a "Forbidden" message from the apache server.
I am using A1 Sitemap Generator 3.0.4. In older versions I had no problems with these directories, but now get a R.Code 403 and a R.Desc Forbidden when I analyze a website. Note: There are other files in the directory which A1 handles correctly.
This is fine but I do not want these error urls included in the sitemaps, because it causes errors when search engine crawl such a site using the generated sitemap.xml..
How do I tell A1 Sitemap Generator to ignore these error in the generated sitemap?

Webhelpforums

While A1 Sitemap Generator will show you URLs that gave 403 response (forbidden) errors, it won't actually include them in the actual generated sitemap files. See: http://www.microsystools.com/products/sitemap-generator/help/sitemap-website-scan-errors/

There was/is a special corner case when creating HTML sitemaps with errored-directory-URLs that contain non-errored-URLs. But I don't think that is what you are referring to?
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

Troffel

Thanks for your prompt reply:
"it won't actually include them in the actual generated sitemap files"
This is my problem - the generated sitemap does include the error urls.

Webhelpforums

And you have recrawled your website + re-created sitemap file after you made the URLs respond with 403?

If so, please email me your project file + URL example of 403 that you believe A1SG includes in generated sitemap files, and I will be happy to take a look today/tomorrow! :)
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

Troffel

Thank you. This issue has been SOLVED!
I have recrawled the website (after i marked 'Always scan directories that contain linked URLs') + re-created sitemap file (I made the directory URLs without index.html file to respond with 403 a long time ago).

Webhelpforums

Good to hear problem has been resolved! :)

By the way, why don't you redirect 301
example.com/dir/index.html to example.com/dir/ ?

I think that is more common than the other way around!

Happy New Year!
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

Troffel


More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
      
website analysis spider tool for technical SEOA1 Website Analyzer
      
SEO tools for managing keywords and keyword listsA1 Keyword Research
      
complete website copier toolA1 Website Download
      
create custom website search enginesA1 Website Search Engine
      
scrape data into CSV, SQL and databasesA1 Website Scraper