Webmaster Forums - Website and SEO Help

Search Engines and Marketing => Website Crawling and Indexing => Topic started by: ferriesthai on September 20, 2019, 02:01:41 AM

Title: How to Block folder and file through Roots.txt File?
Post by: ferriesthai on September 20, 2019, 02:01:41 AM
How to Block folder and file through Roots.txt File?
Title: Re: How to Block folder and file through Roots.txt File?
Post by: Raj on September 24, 2019, 09:06:22 AM
In robots.txt file, simply use the disavow tag. Write down the URL of the file which you want to block just in front of Disavow tag. Same thing with the directory path, just place the directory path in front of Disavow tag. In this way you can block the things.
Title: Re: How to Block folder and file through Roots.txt File?
Post by: saisaanvi on September 25, 2019, 10:15:48 AM
Just mention disallow:/URL to the folder which you want to block. Then Google and other search engine bots won't crawl that particular folder