Hi,
Why should you care about robots.txt file?

Reasons you may Need a Robots.txt File: Following are some reasons because of which your website may need a robots.txt file: Your website has some content that you don't want to be indexed by search engines. Your website contains certain paid links or ads that have special instructions for robots.

The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots which pages on your site to crawl. It also tells web robots which pages not to crawl.