Webmaster Forums - Website and SEO Help

Microsys Products and Webmaster Tools => A1 Sitemap Generator => Topic started by: tphughes on March 25, 2013, 02:16:17 PM

Title: Excluding a folder and file type
Post by: tphughes on March 25, 2013, 02:16:17 PM
I want to exclude certain pages from being indexed. However, it is an AND statement. I want to exclude all pages that are in the /products/ folder AND have the extension of .html. How would I do this?
Title: Re: Excluding a folder and file type
Post by: Webhelpforums on March 25, 2013, 02:47:39 PM
You will want to use regular expressions in those case.

First see:

http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/ (http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/)
(which URLs/pages get analyzed)

http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/ (http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/)
(which URLs/pages get included in output)

Other help pages
http://www.microsystools.com/products/sitemap-generator/help/easy-sitemap-generator-mode/ (http://www.microsystools.com/products/sitemap-generator/help/easy-sitemap-generator-mode/)
http://www.microsystools.com/products/sitemap-generator/help/sitemap-generator-user-interface/ (http://www.microsystools.com/products/sitemap-generator/help/sitemap-generator-user-interface/)

Then create a regular expression like this
Code: [Select]
/products/.*\.htmlwhich will match any URL with */products/* and *.html*

As seen from help above, A1SG needs regex filters prepended with "::" so it becomes
Code: [Select]
::/products/.*\.html