Website Analyser - Not crawling the page

  • 1 Replies
  • 975 Views
*

Vinosrhythm

  • Newbie
  • *
  • 1
  • +0/-0
    • View Profile
Website Analyser - Not crawling the page
« on: January 25, 2016, 12:30:04 PM »
Hi Team,
We are experiencing some issues while running crawler for specific URL. When I tried to run the crawler at domain level , there were some URLs which were reported with -1 and -8 as response code.

When I try to run the same URL separately , I am getting 404. there is a "/" appended at the end of the URL. the website that we are trying to crawl does not have any extension appened at the end. all the urls are without extension.

Can you please suggest how to run crawler for specific URL.

Need some guidance on it , so please provide asap.


*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1379
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
Re: Website Analyser - Not crawling the page
« Reply #1 on: January 26, 2016, 07:05:02 AM »
From here:
http://www.microsystools.com/products/website-analyzer/help/server-http-response-codes/

There you can see -8 simply means the url uses canonical to point to another url

And -1 that it means the url was not checked by the crawler

As such, both are benign information codes.

However, you can switch off this option:
""Scan website | Crawler options" - Use special response codes for when page URLs use canonical or similar"

This is also default in new projects in recent A1WA versions since the state flags visible in he data columns provide even more detailed information



...

If you only want oo run the crawler for a specific/single URL see:
http://www.microsystools.com/products/website-analyzer/help/crawl-website-pages-list/


MicrosysTools.com | Website and SEO Software for webmasters | A1 Sitemap Generator, A1 Website Analyzer etc.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper