Hello,
I have a website which has (among other things) a large list of items, lets say 150, that the user can sort or filter
using a number of options. The filter/sort options that the user selects is then passed in the url as a query string
to reduce the list and only display the items matching the criteria.
When I run the sitemap generator, it creates mappings for every filter option and therefore lists many of the items
numerous times because they would show up in more than one of the filtering/sort-by selections.
My question is, Should I leave the map this way with a lot of duplicated content ? or is there a way to avoid the sitemap generator from following links generated by the filtering / sort-by options(buttons) ?
Thanks in advance for any help ! :)
If your website uses canonical, A1SG will know which URLs to exclude when building the actual sitemap files:
http://www.microsystools.com/products/sitemap-generator/help/sitemap-robots-noindex-nofollow/ (http://www.microsystools.com/products/sitemap-generator/help/sitemap-robots-noindex-nofollow/)
Otherwise you can use "output filters":
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/ (http://www.microsystools.com/products/sitemap-generator/help/website-crawler-output-filters/)
(To ensure the output after scan / generated sitemaps are correct)
To save crawler/analysis time and resources, use "crawler filters":
http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/ (http://www.microsystools.com/products/sitemap-generator/help/website-crawler-scanner-filters/)