How to Submit Robot.txt File In Webmaster..?

Started by Sanusaini, October 11, 2017, 01:56:44 AM


virginoilseom

1. Identify which directories and files on your web server you want to block from being crawled by the crawler
2. Identify whether or not you need to specify additional instructions for a particular search engine bot beyond a generic set of crawling directives
3. Use a text editor to create the robots.txt file and directives to block content
4. Optional: Add a reference to your sitemap file (if you have one)
5. Check for errors by validating your robots.txt file
6. Upload the robots.txt file to the root directory of your site

For more info : https://www.bing.com/webmaster/help/how-to-create-a-robots-txt-file-cb7c31ec

satheesh

Download your edited robots.txt code from the robots.txt Tester page by clicking Download in the Submit dialog. 3. Upload your new robots.txt file to the root of your domain as a text file named robots.txt (the URL for your robots.txt file should be /robots.txt ).

sinelogixtech

Hi Friends,
The robots.txt does not need to be submitted manually to the Search Console (if that's what you meant by "submitting to Google).

All Search Engines look for this file by default in the root of your domain. You can either create it manually(use Notepad for this) and upload it to your server.

If your website runs on Wordpress, you are most likely using the Yoast SEO plugin. In this case you can use it create it (Yoast –> Tools –> File Editor –> Robots.txt). The file will automatically be accessible to search engines from there.

More About Our Webmaster Tools for Windows and Mac

HTML, image, video and hreflang XML sitemap generatorA1 Sitemap Generator
      
website analysis spider tool for technical SEOA1 Website Analyzer
      
SEO tools for managing keywords and keyword listsA1 Keyword Research
      
complete website copier toolA1 Website Download
      
create custom website search enginesA1 Website Search Engine
      
scrape data into CSV, SQL and databasesA1 Website Scraper