What is a web crawler?
A web crawler is a program that automatically visits web pages and retrieves their content. It is also commonly called a bot.
Search engines use crawlers such as Googlebot to gather pages for search. AI providers also document agents such as OpenAI's GPTBot and Anthropic's ClaudeBot. A request identifies a user agent, and sites can publish rules for named agents in robots.txt.
The provider name does not describe every use
A provider may distinguish crawling for model development, crawling for search and retrieving a page at a user's request. Blocking one named agent does not necessarily stop the others.
Match the control to the purpose you intend to allow or restrict. Consult the provider's documented agent names and the scope of its rules rather than treating all requests from one company as interchangeable.
Where to look when a crawler cannot fetch a page
Review both the site's published rules and its CDN or firewall protections. A permissive robots.txt file cannot make a blocked network request succeed.
Also distinguish fetching from appearing in search results. noindex concerns indexing, whereas a crawler's ability to retrieve the page concerns access. The distinction matters when diagnosing a missing page.
Sources and scope
- OpenAI: crawler documentation
- Anthropic: web crawling and site-owner controls
- Google Search Central: common crawlers
Source-language entry records a source check on 2026-09-26. This adaptation preserves that provenance and does not represent a new audit of the cited pages.