I hope, you all must listen about Crawl delay. Lots of webmaster use "crawl-delay" word in the robots.txt file. I searched about this, on Google and got lots answer. But, I did not get any answer, which satisfied to me. So, I made topics on this and posted here.
Sometimes, you have seen this Line in robots.txt file.
User-agent: Googlebot
Crawl-delay: 10
Can someone explain? What is the meaning above line?
User-agent: Googlebot
Crawl-delay: 10
Craw- delay is the directive used in website where continuous content update. Suppose in twitter, Googlebot and other bots continuous visit this website frequently to get real time update. Now if this bots continue visit this site then it creates load on server. So this kind of website use crawl-delay directive to setting out interval for each bots.