only crawl and show robot allowed pages + show source of duplicated pages links.

Started by togfather, December 26, 2016, 06:46:34 PM

togfather

Hello. 

Is there a tutorial for this software?

How do I get a1 website analyser to stop crawling pages disallowed in the robots.txt file?

Also, when listing pages with identical titles, can I get it to show me which pages link to the duplicated pages?

Many thanks

Tog

Webhelpforums

A1 Website Analyzer will not per default crawl pages disallowed in robots.txt

However, it will show them post scan since they were discovered.  (They will have a flag, so you can see the URLs were disallowed in robots.txt)

This behavior is configurable. If you want to have such URLs removed, before scan enable option:
Scan website | Crawler options | Apply "webmaster" and "output filters" after website scan stops.

See also:
http://www.microsystools.com/products/website-analyzer/help/crawl-robots-noindex-nofollow/

...

To get started using A1 Website Analyzer see:
http://www.microsystools.com/products/website-analyzer/help/site-analysis-seo-audit/

Search Engine People also written various tutorials - listed here among other guides written by users:
http://www.microsystools.com/products/website-analyzer/help/seo-analysis-guides/

TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

togfather

Thank you very much.

That was very helpful and I have now got the pages covered by robots.txt removed.

best wishes

Tog

More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
      
website analysis spider tool for technical SEOA1 Website Analyzer
      
SEO tools for managing keywords and keyword listsA1 Keyword Research
      
complete website copier toolA1 Website Download
      
create custom website search enginesA1 Website Search Engine
      
scrape data into CSV, SQL and databasesA1 Website Scraper