what is the difference between crawling and Indexing?

Started by sonyrobin, May 01, 2018, 08:08:14 AM


mvminfotech

A web crawler (also known as a web spider or web robot) is a program or automated script which browses the World Wide Web in a methodical, automated manner. This process is called Web crawling or spidering. Many legitimate sites, in particular search engines, use spidering as a means of providing up-to-date data.

Indexing is the processing of the information gathered by the Googlebot from its crawling activities. Once documents are processed, they are added to Google's searchable index if they are determined to be quality content.

joelhuang

Crawling or spidering is a term used when Google, or another search engine, sends its bot to a web page or web post and "reads" the page.  Don't let this be confused with having that page being indexed.  Crawling is the first part of having a search engine recognize your page and show it in search results.  Having your page crawled does not necessary mean your page was indexed and will be found.  Pages get crawled for a variety of reasons...the most common is having an XML sitemap that Google reads, which points to your new page.

Indexing means making sense out of the retrieved contents, storing the processing results in a (more or less complex) document index. Link analysis is a way to measure URI importance, popularity, trustworthiness and so on. Link analysis is often just a helper within the indexing process, sometimes the end in itself, but traditionally a task of the indexer, not the crawler (high sophisticated crawling engines do use link data to steer their crawlers, but that has nothing to do with link analysis in document indexes).

Krina

 Crawling means that Googlebot looks at all the content/code on the page and analyzes it.
Indexing means that the page is eligible to show up in Google's search

RH-Calvin

Crawling is the process or reading through your webpage source by search engine spiders. They provide a cache certificate after a successful crawl. Indexing is updating the cached webpages in search engine database. Indexed webpages are now ready for search engine rankings.
Cheap VPS | $1 VPS Hosting
Cheap Dedicated Servers | Free Setup with IPMI