crawler denied access to my site badbot trap

  • 4 Replies
  • 2100 Views
*

grizzlyff

  • Newbie
  • *
  • 2
  • +0/-0
    • View Profile
crawler denied access to my site badbot trap
« on: February 04, 2012, 09:05:54 AM »
Hey, I am trying your sitemap software and i really like it so far, and I am sure I will be buying it, but I have the inevitable newbie question:


I set it up, and was working no problem, then I tried it on a site of mine that has a bad bot trap that excludes any crawler or bot except ones that I specifically allow....I could allow it by ip address but my IP address changes sometimes so would prefer to allow it by name, so could you tell what name to use, or is there a field that I could configure to a certain name, so I can allow it?

*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1413
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
Re: crawler denied access to my site badbot trap
« Reply #1 on: February 04, 2012, 12:47:42 PM »
Sure, there is an entire help page devoted to help people get past such problems! :)

http://www.microsystools.com/products/sitemap-generator/help/sitemap-generator-website-platform/

Essentially, just change the user agent crawler ID at "General options | Internet crawler"
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

*

grizzlyff

  • Newbie
  • *
  • 2
  • +0/-0
    • View Profile
Re: crawler denied access to my site badbot trap
« Reply #2 on: February 04, 2012, 03:42:44 PM »
thanks......I tried that, and also just making a directive for the miggibot, but the issue i still have is that your software is still getting trapped as a "bad" bot..........I will look at my robots.txt, but it looks like it either is following the link set to no follow, or I have the robots set wrong, i will look at it some more before i bother u about it

*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1413
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
Re: crawler denied access to my site badbot trap
« Reply #3 on: February 05, 2012, 07:00:48 AM »
Hi,

Please drop an email to support with your website address
http://www.microsystools.com/home/contact.php

It should not be a problem, but easier to help solve when seeing and testing on the website!
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

*

Webhelpforums

  • Administrator
  • Hero Member
  • *****
  • 1413
  • +6/-0
  • Shared between Microsys, WebHelpForums and helpers
    • View Profile
    • Webmaster and Website Help Forums
Re: crawler denied access to my site badbot trap
« Reply #4 on: March 03, 2012, 11:51:28 AM »
Problem was resolved over email quite some time ago. (A1SG had not been allowed in robots.txt, wrong user agent string was used in robots.txt)
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper