What is Claude-SearchBot?

How Anthropic describes it

In the help center article covering its web bots, Anthropic describes Claude-SearchBot as navigating the web to improve search result quality for users, analyzing online content specifically to enhance the relevance and accuracy of search responses. Disabling it prevents Anthropic's system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in Claude's AI search results.

Two things are worth noticing. Anthropic does not publish a full user-agent string for this bot; the name to use in robots.txt is Claude-SearchBot. And the article says nothing about how sources are chosen or ranked inside Claude's answers. Being reachable is a precondition for being cited, not a promise of it.

Indexing ahead versus fetching on demand

AI assistants get information from the web in two ways. In one, a bot visits pages in advance and builds an index to draw on at search time; for Anthropic, that is Claude-SearchBot's job. In the other, a page is fetched at the moment a user asks; that belongs to Claude-User. OpenAI draws the same line between OAI-SearchBot and ChatGPT-User.

The practical difference: a page that was never indexed ahead of time has a harder time surfacing as a candidate in a search, unless the user names it or pastes its link. Anthropic does not explain in detail how the two bots interact, so it is wise not to assume a precise mechanism, and equally unwise to treat one as a substitute for the other.

Subdomains and crawl rate

Anthropic says its bots honor standard "do not crawl" directives in robots.txt, and that a block needs to be applied on every subdomain you want to opt out. Because robots.txt is read per host, a rule on example.com does nothing for portal.example.com. Staying open to search on the main site while closing a customer portal takes two files:

# https://example.com/robots.txt
User-agent: Claude-SearchBot
Allow: /
Disallow: /cart/

# https://portal.example.com/robots.txt
User-agent: Claude-SearchBot
Disallow: /

Anthropic also supports the non-standard Crawl-delay extension. If server load is the worry, spacing requests out can ease it while keeping search visibility intact, which an outright block would not.

Before you block it

• Training is a separate switch. If you do not want content used for model training, the relevant bot is ClaudeBot. Blocking Claude-SearchBot does nothing for that goal and only lowers search visibility.
• IP blocks are discouraged. Anthropic warns that blocking its IP addresses may not work correctly or guarantee a lasting opt-out, because it also stops its bots from reading your robots.txt. For verifying source addresses it publishes https://claude.com/crawling/bots.json.
• Challenge pages block silently. Anthropic says its bots will not try to bypass CAPTCHAs. If your firewall serves a challenge to unfamiliar traffic, the bot never reaches the content even though robots.txt allows it. Those layers are covered under bot accessibility.