I've just done an initial test scan on our website. It's picked up A LOT of pages, but seems to be missing certain product pages.
For example, a page like www.ourwebsite.com/Product-Category.aspx?prodid=1775 doesn't seem to have been picked up in the scan. Through the website, this product page would be brought up by doing a search for a specific part number, then clicking on the link to the product page.
How do I go about getting these kinds of pages picked up during the website scan?
Any advice would be great, thanks.
Hi,
If possible your site should have all pages interlinked, so the crawler in A1SG can discover them.
Alternatively, you can import a list of URLs and enable *Recrawl* setting. (Doing that, the rest of the site will also be crawled.)
http://www.microsystools.com/products/sitemap-generator/help/website-sitemap-pages-list/ (http://www.microsystools.com/products/sitemap-generator/help/website-sitemap-pages-list/)
If just a few missing URLs, you can also add those URLs to start search paths:
http://www.microsystools.com/products/sitemap-generator/help/root-aliases-start-paths/ (http://www.microsystools.com/products/sitemap-generator/help/root-aliases-start-paths/)
You can also try enable Javascript parsing in "Scan website | Crawler options" if you use Javascript.
Thank you for this information, it's given us a great start.
Analysing the structure of the site suggests interlinking may be the biggest issue, but the other suggestions are just as helpful.
Thanks again.
We're getting some really good results with this software.
Another quick question. With this software, is it possible to create a robots.txt file to allow certain URLs to be crawled but not indexed (via filtering)? Or is it only possible to prevent URLs from being crawled at all?
You can not differ between "nofollow" and "noindex" when using robots.txt (not supported by the robots.txt format)