what is difference between spider, crawler and robots?

Started by Black Ninja, March 05, 2012, 01:32:34 AM

Black Ninja

What is difference between spider, crawler and robots? Or Spider, crawler and robots is same search software to craw and index webpage for the search engine.

M. Rebecca

300% more Adsense earnings with 12+ Keyword Research Tools

shaswata

Yes, basically they all are same & used to crawl & index webpages for different search engines.

pawpoint

They are just different names for the same thing. "Spiders" are "Crawlers" and "Robots" is a word commonly used when talking of robot text which is where you put exclusions for the spiders not to find.

spinxwebdesign

Spider, crawler  and robots both are same program to reads the text information from World Wide Web and find hyperlinks if any, then followed those links and collect all data and records it into search engine database for use of visitors.

Bitoo

Spider is a application program used to scan the web pages from website.

Crawler is used to crawl or index the pages from website.

Robot is a text file used for access permission to spider and crawler.

Nichole Green

Crawler or Spider, a robot is a search engine program that crawls the web, collecting data, following links, making copies of new and updated sites, and storing URLs in the search engine's Index. This allows search engines to provide faster and more up-to-date listings.

tarunjangra

spiders, crawler and robots are just different names for the same thing.these are different words to define the automated programs which are written to do crowling through the web. The crawlers are automated so they are called Robots and they are crawling the web (internet) so they are called Spiders.
Robots do the predefined tasks which are to download we pages and follow the links on those pages.



mariyaceron

Thanks for sharing the difference between Spider,Crawler & Robot.This is really beneficial information for me.
http://www.tier1automation.com/c-4-motor_control.aspx
motor control

ericfox

According to me spider, crawler and robots is same search software.  All works for search engine like traveling the web and find out the new update or information and links and submit it in search engine data base.

siyajoshi

Spider - a browser like program that downloads web pages.

Crawler – a program that automatically follows all of the links on each web page.

Robots - An automated computer program that visit websites & perform predefined task.

jennifer1234

Spider, crawler and robots all are same program to reads the text information from World Wide Web.

apsarabanu

A program that automatically fetches Web pages. Spiders are used to feed pages to search engines. It's called a spider because it crawls over the Web.

A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion.

A robot is a mechanical or virtual intelligent agent that can perform tasks automatically or with guidance, typically by remote control.

icecube media

Hello,

Spider Scan and judge you website content ,Crawler crawl you content and index it in there searches according to Inbound links both techniques helps google to judge importance of website in terms of ranking ,robot.txt file is that file which is included into web coding so that google not consider it in crawling or to hide page from google crawling .

More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
      
website analysis spider tool for technical SEOA1 Website Analyzer
      
SEO tools for managing keywords and keyword listsA1 Keyword Research
      
complete website copier toolA1 Website Download
      
create custom website search enginesA1 Website Search Engine
      
scrape data into CSV, SQL and databasesA1 Website Scraper