crawler denied access to my site badbot trap

Started by grizzlyff, February 04, 2012, 09:05:54 AM

grizzlyff

Hey, I am trying your sitemap software and i really like it so far, and I am sure I will be buying it, but I have the inevitable newbie question:


I set it up, and was working no problem, then I tried it on a site of mine that has a bad bot trap that excludes any crawler or bot except ones that I specifically allow....I could allow it by ip address but my IP address changes sometimes so would prefer to allow it by name, so could you tell what name to use, or is there a field that I could configure to a certain name, so I can allow it?

Webhelpforums

Sure, there is an entire help page devoted to help people get past such problems! :)

http://www.microsystools.com/products/sitemap-generator/help/sitemap-generator-website-platform/

Essentially, just change the user agent crawler ID at "General options | Internet crawler"
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

grizzlyff

thanks......I tried that, and also just making a directive for the miggibot, but the issue i still have is that your software is still getting trapped as a "bad" bot..........I will look at my robots.txt, but it looks like it either is following the link set to no follow, or I have the robots set wrong, i will look at it some more before i bother u about it

Webhelpforums

Hi,

Please drop an email to support with your website address
http://www.microsystools.com/home/contact.php

It should not be a problem, but easier to help solve when seeing and testing on the website!
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

Webhelpforums

Problem was resolved over email quite some time ago. (A1SG had not been allowed in robots.txt, wrong user agent string was used in robots.txt)
TechSEO360 | MicrosysTools.com  | A1 Sitemap Generator, A1 Website Analyzer etc.

More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
      
website analysis spider tool for technical SEOA1 Website Analyzer
      
SEO tools for managing keywords and keyword listsA1 Keyword Research
      
complete website copier toolA1 Website Download
      
create custom website search enginesA1 Website Search Engine
      
scrape data into CSV, SQL and databasesA1 Website Scraper