Robots.txt is a file that orders the search engine crawlers to determine which URLs they can reach on the website or which can not. It is specially used to prevent the overloading of the site the requests and maintain the crawler traffic. It gives a set of rules. It is...