Use of robot.txt ?

  • 10 Replies
  • 760 Views
*

bangalorewebguru

  • Website Design & Development Company in Bangalore
  • Sr. Member
  • ****
  • 288
  • +0/-0
  • Making Smartwork
    • View Profile
    • Web Designing Company in Bangalore,India
Use of robot.txt ?
« on: June 08, 2017, 04:17:02 AM »
Use of robot.txt ?
Web Designing Company Bangalore | Web Designing Companies Bangalore | Website Development Company Bangalore

*

RH-Calvin

  • Hero Member
  • *****
  • 1080
  • +0/-0
    • View Profile
    • Reprise Hosting
Re: Use of robot.txt ?
« Reply #1 on: June 12, 2017, 09:51:47 AM »
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Cheap VPS | $1 VPS Hosting
Cheap Dedicated Servers | Free Setup with IPMI

*

sirishasiri

  • Full Member
  • ***
  • 107
  • +0/-0
    • View Profile
Re: Use of robot.txt ?
« Reply #2 on: December 06, 2017, 06:56:37 AM »
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard
used by websites to communicate with web crawlers and other web robots. The standard specifies how to
inform the web robot about which areas of the website should not be processed or scanned.

*

compressjpg

  • Newbie
  • *
  • 4
  • +0/-0
    • View Profile
Re: Use of robot.txt ?
« Reply #3 on: December 14, 2017, 09:44:29 AM »
Thanks for sharing useful information...

*

hotmailserviceuk

  • Hotmail Technical Support Number
  • Newbie
  • *
  • 25
  • +0/-0
    • View Profile
    • Hotmail Contact Number
Re: Use of robot.txt ?
« Reply #4 on: December 20, 2017, 02:03:03 AM »
Hello Guys
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots.

*

salenaadam

  • Full Member
  • ***
  • 111
  • +0/-0
    • View Profile
Re: Use of robot.txt ?
« Reply #5 on: February 09, 2018, 01:07:30 AM »
Robots.txt is a text  file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. Robots.txt is a text file located in the site’s root directory that specifies for search engines’ crawlers and spiders what website pages and files you want or don’t want them to visit.

*

praveenitech1

  • Full Member
  • ***
  • 124
  • +0/-0
    • View Profile
Re: Use of robot.txt ?
« Reply #6 on: February 21, 2018, 02:51:14 AM »
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

*

vishnu priya

  • Phone repair and service shop
  • Full Member
  • ***
  • 142
  • +0/-0
  • Smartphone Screen replacement shop
    • View Profile
    • Samsung service and repair shop
Re: Use of robot.txt ?
« Reply #7 on: March 07, 2018, 07:26:29 AM »
Robots.txt is a text file created to instruct search engine robots, how to crawl pages on their website.

*

agustina

  • Jr. Member
  • **
  • 62
  • +0/-0
    • View Profile
    • International MBA Chennai
Re: Use of robot.txt ?
« Reply #8 on: March 17, 2018, 02:25:43 AM »
Web owners only have rights to use robot.txt file to provide information about their site to web robots

*

grawhill

  • Newbie
  • *
  • 31
  • +0/-0
    • View Profile
    • Android application developers Dubai
Re: Use of robot.txt ?
« Reply #9 on: March 23, 2018, 06:05:07 AM »
A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’.  For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.

*

digitalaimschool

  • SEO Training Institute in Faridabad
  • Newbie
  • *
  • 8
  • +0/-0
  • Internet Marketing Training Institute
    • View Profile
    • Digital Marketing Training Institute in Faridabd
Re: Use of robot.txt ?
« Reply #10 on: March 27, 2018, 06:58:09 AM »
Robots.txt is a text file that lists site pages and using robot.txt file you can instruct search engines robots. Which webpage are allowed and disallowed for crawling and  It contain instructions for robots.

 




See Our Webmaster Tools for Windows and Mac

A1 Sitemap Generator
      
A1 Website Analyzer
      
A1 Keyword Research
      
A1 Website Download
      
A1 Website Search Engine
      
A1 Website Scraper