Glossary / C
Crawler
In one line
A bot that discovers and reads web pages.
Crawlers such as Googlebot follow links and fetch pages so they can be indexed. You guide them with internal links, XML sitemaps and robots.txt.
Also called
Spider, Bot