Hello friends, if you have a website and wants search engine not crawl or index some file or directory from your website then with the help of robots.txt you can do it. As you understand from robots.txt file extension it’s a simple text document file which is used to instruct search engine how to crawl or index web pages of their website. You must upload this robots.txt file in main directory (root) of hosting server otherwise search engine not able to search and read this file.
Syntax of Robots.txt: Basic structure of robots.txt file is endless combination of “user-agent” & “disallow” file and directories
Here “User-agent” stands for search engine crawlers/bots or “Disallow:” used to include file name and path of directories which you want to exclude from indexing.
There different number of combination for robots.txt to give different command to search engines which are as follow:
Block all search engine crawlers/bots from whole website:
Block a particular search engine crawlers/bots from particular directory/folder:
Block a particular search engine crawlers/bots from particular file or web page: