Website Analyser - Not crawling the page

Started by Vinosrhythm, January 25, 2016, 12:30:04 PM


Hi Team,
We are experiencing some issues while running crawler for specific URL. When I tried to run the crawler at domain level , there were some URLs which were reported with -1 and -8 as response code.

When I try to run the same URL separately , I am getting 404. there is a "/" appended at the end of the URL. the website that we are trying to crawl does not have any extension appened at the end. all the urls are without extension.

Can you please suggest how to run crawler for specific URL.

Need some guidance on it , so please provide asap.


From here:

There you can see -8 simply means the url uses canonical to point to another url

And -1 that it means the url was not checked by the crawler

As such, both are benign information codes.

However, you can switch off this option:
""Scan website | Crawler options" - Use special response codes for when page URLs use canonical or similar"

This is also default in new projects in recent A1WA versions since the state flags visible in he data columns provide even more detailed information


If you only want oo run the crawler for a specific/single URL see:

TechSEO360 |  | A1 Sitemap Generator, A1 Website Analyzer etc.

More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
website analysis spider tool for technical SEOA1 Website Analyzer
SEO tools for managing keywords and keyword listsA1 Keyword Research
complete website copier toolA1 Website Download
create custom website search enginesA1 Website Search Engine
scrape data into CSV, SQL and databasesA1 Website Scraper