Robot templates allows you to specify which pages should not be crawled. Pages that don't get crawled can still rank for keywords and show up in search results. Robot templates has been with us for over 14 years, but how many of us knew that in addition to the disallow directive there's a noindex directive that Googlebot obeys? That noindexed pages don't end up in the index but disallowed pages do, and the latter can show up in the search results (albeit with less information since the spiders can't see the page content). It is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
It contains restrictions for Web Spiders, telling them where they have robot templates permission to search. It is like defining rules for search engine spiders (robots) what to follow and what not to. It provides you with more functionality than Meta robots tag which is available only partially to control behaviour of search engines. You can use it to prevent indexing totally, prevent certain areas of your site from being indexed or to issue individual robot templates indexing instructions to specific search engines. Robot.txt protocols are simply advisory though. There is no law requiring websites to have Robot.txt files, or to use them on their web pages.
It is the most widely used method for controlling the behaviour of automated robot templates on your site (all major robots, including those of Google, Alta Vista, etc. It can be used to block access to the whole domain, or any file or directory within.
It is a text file which instructs search engine spiders or crawlers on what to do robot templates. It tells specific web spiders on which specific web pages to index robot templates. Robots are configured to read text. Too much graphic content could render your pages invisible to the search engine. Robot Manager uses a simple user interface that makes creating your robot templates file a breeze.
They can come in very handy beyond the search engines. It is possible to use them to protect your site from malevolent web crawlers, which is useful to say the least.