Picking up dynamic aspx? product pages during website scan

  • 4 Replies
  • 2046 Views
*

Laptop Plus

  • Newbie
  • *
  • 7
  • +0/-0
    • View Profile
Picking up dynamic aspx? product pages during website scan
« on: November 12, 2013, 12:46:22 AM »
I've just done an initial test scan on our website. It's picked up A LOT of pages, but seems to be missing certain product pages.

For example, a page like www.ourwebsite.com/Product-Category.aspx?prodid=1775 doesn't seem to have been picked up in the scan. Through the website, this product page would be brought up by doing a search for a specific part number, then clicking on the link to the product page.

How do I go about getting these kinds of pages picked up during the website scan?

Any advice would be great, thanks.

*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1413
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
Re: Picking up dynamic aspx? product pages during website scan
« Reply #1 on: November 12, 2013, 05:45:52 AM »
Hi,

If possible your site should have all pages interlinked, so the crawler in A1SG can discover them.

Alternatively, you can import a list of URLs and enable *Recrawl* setting. (Doing that, the rest of the site will also be crawled.)
http://www.microsystools.com/products/sitemap-generator/help/website-sitemap-pages-list/

If just a few missing URLs, you can also add those URLs to start search paths:
http://www.microsystools.com/products/sitemap-generator/help/root-aliases-start-paths/

You can also try enable Javascript parsing in "Scan website | Crawler options" if you use Javascript.

TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

*

Laptop Plus

  • Newbie
  • *
  • 7
  • +0/-0
    • View Profile
Re: Picking up dynamic aspx? product pages during website scan
« Reply #2 on: November 13, 2013, 01:21:46 AM »
Thank you for this information, it's given us a great start.

Analysing the structure of the site suggests interlinking may be the biggest issue, but the other suggestions are just as helpful.

Thanks again.

*

Laptop Plus

  • Newbie
  • *
  • 7
  • +0/-0
    • View Profile
Re: Picking up dynamic aspx? product pages during website scan
« Reply #3 on: November 28, 2013, 12:16:45 AM »
We're getting some really good results with this software.

Another quick question. With this software, is it possible to create a robots.txt file to allow certain URLs to be crawled but not indexed (via filtering)? Or is it only possible to prevent URLs from being crawled at all?

*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1413
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
Re: Picking up dynamic aspx? product pages during website scan
« Reply #4 on: December 09, 2013, 06:17:18 AM »
You can not differ between "nofollow" and "noindex" when using robots.txt (not supported by the robots.txt format)
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper