Webmaster Forums - Website and SEO Help

Search Engines and Marketing => Sitemaps => Topic started by: anandiphone on September 02, 2011, 08:39:39 AM

Title: What is robots.txt file and use?
Post by: anandiphone on September 02, 2011, 08:39:39 AM
Kindly tell me the actual use of robots.txt file and its use?
Title: Re: What is robots.txt file and use?
Post by: Webhelpforums on September 03, 2011, 04:30:01 PM
To limit search engines and other crawlers from crawling URLs you do not want. (You can also e.g. control crawl-delay to prevent nice-behaving crawler robots from overloading your website server)

You can also use it to point to an XML sitemap
Title: Re: What is robots.txt file and use?
Post by: evelyn8411 on September 04, 2011, 07:50:48 PM
Robot text is used to direct search engines about where they can crawl your site. Of course, only approved SE follow this. Vicious crawlers will not follow your robot text and crawl everything it wants/
Title: Re: What is robots.txt file and use?
Post by: John23 on September 05, 2011, 01:33:28 AM
In point of view Robots txt files are use to give permission to search engine weather to index the webpaeg or foolow the page.there are several instruction for it.
Title: Re: What is robots.txt file and use?
Post by: alvypetersen on September 05, 2011, 02:39:53 AM
Robots.txt is a text file that tells search engine spiders, also known as search engine robots, which parts of your website they can enter and which parts they can’t.
Title: Re: What is robots.txt file and use?
Post by: dpfocanil on September 06, 2011, 05:22:04 AM
A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. It divided into sections by the robot crawler's User Agent name.

User-agent: * Disallow:
Title: Re: What is robots.txt file and use?
Post by: hannahraasch on September 06, 2011, 06:51:27 AM
Simply, the robots.txt is a very simple text file that is placed on our root directory. For example www.example.com/robots.txt. This file tells search engine and other robots which areas of our site they are allowed to visit and index.
Title: Re: What is robots.txt file and use?
Post by: ykgforum on October 18, 2011, 07:34:20 AM
For google page crawler
Title: Re: What is robots.txt file and use?
Post by: Nichole Green on October 25, 2011, 06:00:06 AM
The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Most websites have files and folders that are not relevant for search engines (like images or admin files) therefore creating a robots.txt file can actually improve your website indexation.
A robots.txt is a simple text file that can be created with Notepad. If you are using WordPress a sample robots.txt file would be:
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
If you are not using WordPress just substitute the Disallow lines with files or folders on your website that should not be crawled, for instance:
User-agent: *
Disallow: /images/
Disallow: /cgi-bin/
Disallow: /any other folder to be excluded/
Title: Re: What is robots.txt file and use?
Post by: Black Ninja on November 02, 2011, 02:20:24 AM
Robots.txt allows you to specify which pages should not be crawled. Pages that don't get crawled can still rank for keywords and show up in search results.
Title: Re: What is robots.txt file and use?
Post by: infoanil on November 02, 2011, 03:08:01 AM
Robots.txt is a file that is used to leave out content from the crawling process of search engine spiders / bots. WebPages are indexed by the search engines. But there may be some content that we don’t want to be crawled & indexed. The main plan is we don’t want them to be indexed.
Title: Re: What is robots.txt file and use?
Post by: LucieJason on November 28, 2011, 02:02:15 AM
Robots.txt is veryuseful and helpful file. When a crawler crawl a website, it read robots.txt initially. Robots.txt is used widely to instruct the crawler that which page has to be crawled and which is not.