What is Claude-User?
Definition
Claude-User is the user agent Anthropic uses to support Claude users: when someone asks Claude a question, Claude may access websites under this name. According to Anthropic, Claude-User lets site owners control which sites these user-initiated requests can reach. Disabling it prevents content from being retrieved in response to user queries and may reduce visibility in user-directed web search.
Also known as: Claude User agent, Claude-User user agent, Claude-User bot, Anthropic Claude-User
User-triggered, yet governed by robots.txt
Every major AI provider runs a separate user agent that fetches pages on the spot when a user asks something. How those fetchers treat robots.txt varies by provider, and that is precisely what sets Claude-User apart:
| Fetcher | What the provider says about robots.txt |
|---|---|
Claude-User (Anthropic) | Lets site owners control which sites these user-initiated requests can access; Anthropic says its bots honor robots.txt directives. |
ChatGPT-User (OpenAI) | Because actions are initiated by a user, robots.txt rules may not apply. |
Perplexity-User (Perplexity) | Because a user requested the fetch, it generally ignores robots.txt rules. |
In other words, according to Anthropic's help center article, a robots.txt rule for Claude-User also covers fetches made on a user's behalf. Writing the same rule for ChatGPT-User or Perplexity-User does not carry the same assurance, by those providers' own account.
What disabling it does
Anthropic says that disabling Claude-User prevents its system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search. Picture it concretely: someone pastes the link to your product page into Claude and asks which operating systems the product supports. With Claude-User blocked, Claude cannot read the page, so the answer falls back on other sources or on what the model already knew, and your current details may never reach the reply.
This choice is independent of training. A site that does not want its content used for Anthropic's model training can block ClaudeBot while still allowing Claude-User; conversely, closing Claude-User says nothing about training. Each bot is managed under its own name in robots.txt.
Closing off just part of a site
For most sites the sensible setup is neither all-open nor all-closed, but one that keeps user-specific and transactional areas out:
User-agent: Claude-User
Allow: /
Disallow: /customer-portal/
Disallow: /quote-request/
Disallow: /checkout/Keep two details in mind. Once a bot finds a group addressed to its own name, the rules under User-agent: * no longer apply to it, so repeat any paths you want protected. And Anthropic notes that the rule must go into the robots.txt of every subdomain you want to opt out.
Recognizing it in logs
Anthropic's article does not give a full user-agent string for Claude-User, so filter your logs on Claude-User. Since the name can be spoofed, the source IP of a request can be checked against Anthropic's published list at https://claude.com/crawling/bots.json. Because it acts on user requests, Claude-User traffic is irregular and clusters around moments when your pages are drawing interest.
Even with robots.txt allowing it, Claude-User cannot reach content behind a bot protection challenge page, because Anthropic states its bots will not attempt to bypass CAPTCHAs. Anthropic's bot for its search index is Claude-SearchBot, and the one that collects training data is ClaudeBot.