only crawl and show robot allowed pages + show source of duplicated pages links.

Started by togfather, December 26, 2016, 06:46:34 PM



Is there a tutorial for this software?

How do I get a1 website analyser to stop crawling pages disallowed in the robots.txt file?

Also, when listing pages with identical titles, can I get it to show me which pages link to the duplicated pages?

Many thanks



A1 Website Analyzer will not per default crawl pages disallowed in robots.txt

However, it will show them post scan since they were discovered.  (They will have a flag, so you can see the URLs were disallowed in robots.txt)

This behavior is configurable. If you want to have such URLs removed, before scan enable option:
Scan website | Crawler options | Apply "webmaster" and "output filters" after website scan stops.

See also:


To get started using A1 Website Analyzer see:

Search Engine People also written various tutorials - listed here among other guides written by users:

TechSEO360 |  | A1 Sitemap Generator, A1 Website Analyzer etc.


Thank you very much.

That was very helpful and I have now got the pages covered by robots.txt removed.

best wishes


More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
website analysis spider tool for technical SEOA1 Website Analyzer
SEO tools for managing keywords and keyword listsA1 Keyword Research
complete website copier toolA1 Website Download
create custom website search enginesA1 Website Search Engine
scrape data into CSV, SQL and databasesA1 Website Scraper