Glossary / R
Robots.txt
In one line
A file telling crawlers which parts of a site they may access.
Robots.txt sits at your domain root and can disallow crawling of sections. It controls crawling, not indexing — to keep a page out of results, use a noindex tag instead.