Webmaster Forums - Website and SEO Help

Search Engines and Marketing => Website Crawling and Indexing => Topic started by: indiaastro on June 05, 2019, 03:51:21 AM

Title: How to solve robots.txt error ..?
Post by: indiaastro on June 05, 2019, 03:51:21 AM
Hello friends,

How to solve robots.txt error ..?
Title: Re: How to solve robots.txt error ..?
Post by: Rscomp on June 07, 2019, 07:15:30 AM
have you made sure your robots.txt is loading in your browser by adding robots.txt after your domain same as a normal page and can you see contents? has your site been down in this period? have you changed the contents of the file just before this issue? are you sure Googlebot hasn't come back since that date - whats your analytics say? do an index site: search for your domain to see if it is in google.
Title: Re: How to solve robots.txt error ..?
Post by: Jessicad on June 10, 2019, 03:43:16 AM
A robots.txt file is handy for telling search engines which parts of a website should be crawled/indexed and which parts shouldn't. This can be useful in certain situations where you want to keep a page or an asset hidden from search engines.
Title: Re: How to solve robots.txt error ..?
Post by: tanwirjain on June 10, 2019, 05:08:52 AM
Create a new robots.txt file and reupload it.