How to create a Robots.txt file ?
Robot Exclusion Standard Protocol or Robots.txt is a text file which is placed at the root of website. The standard was proposed by Martijn Koster in 1994. The main aim of robots.txt file is to tell web crawler or robots to index or not index a website/webpage.
Read More
