Robots.txt File Explained
Back to Infographics
Robots.txt file simply tells the search engine crawlers the places they are not allowed to index on your site.