Why we use Robots.txt File?
Robots.txt is a file to allow or disallowed search engine to keep some pages privacy..
1- if you don't have robots.txt file for your website, please create one to not crawl your private or confidential directories.
2- Determine any harm by robots.txt, any wrong instruction can block your website from search engines, so please check again your robots.txt file.
3- Determine the need of robots.txt file for your website.
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Robots.txt file is a form of text that contains the code that webmasters create, which binds search engine bots to your website.
The content of a robots.txt file consists of so-called "records". both "/support-desk/index.html" and "/support/index.html" as well as all other files in the "support" directory would not be indexed by search engines. If you leave the Disallow line blank, you're telling the search engine that all files may be indexed.