only crawl and show robot allowed pages + show source of duplicated pages links.

  • 2 Replies
  • 655 Views
*

togfather

  • Newbie
  • *
  • 3
  • +0/-0
    • View Profile
Hello. 

Is there a tutorial for this software?

How do I get a1 website analyser to stop crawling pages disallowed in the robots.txt file?

Also, when listing pages with identical titles, can I get it to show me which pages link to the duplicated pages?

Many thanks

Tog

*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1385
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
A1 Website Analyzer will not per default crawl pages disallowed in robots.txt

However, it will show them post scan since they were discovered.  (They will have a flag, so you can see the URLs were disallowed in robots.txt)

This behavior is configurable. If you want to have such URLs removed, before scan enable option:
Scan website | Crawler options | Apply "webmaster" and "output filters" after website scan stops.

See also:
http://www.microsystools.com/products/website-analyzer/help/crawl-robots-noindex-nofollow/

...

To get started using A1 Website Analyzer see:
http://www.microsystools.com/products/website-analyzer/help/site-analysis-seo-audit/

Search Engine People also written various tutorials - listed here among other guides written by users:
http://www.microsystools.com/products/website-analyzer/help/seo-analysis-guides/

« Last Edit: December 27, 2016, 09:05:47 AM by Webhelpforums »
MicrosysTools.com | Website and SEO Software for webmasters | A1 Sitemap Generator, A1 Website Analyzer etc.

*

togfather

  • Newbie
  • *
  • 3
  • +0/-0
    • View Profile
Thank you very much.

That was very helpful and I have now got the pages covered by robots.txt removed.

best wishes

Tog

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper