What is 429 Too Many Requests?
Definition
429 Too Many Requests is the HTTP status code saying a client has sent too many requests in a given amount of time, in other words that it hit a rate limit. Defined in RFC 6585, the response may include a Retry-After header stating how long to wait. Google's crawlers treat a 429 as a sign that the server is overloaded, count it as a server error and slow down crawling.
Also known as: HTTP 429, 429 error, rate limit exceeded
Retry-After tells clients when to come back
A 429 is the visible face of rate limiting. RFC 6585 says the response should explain the condition and may carry a Retry-After header saying how long to wait. The value is either a number of seconds or an HTTP date:
HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 120
{"error": "rate_limited", "detail": "Max 60 requests per minute"}The standard deliberately leaves open how the server identifies the client or counts requests: the limit can apply per IP, per API key, per account or per endpoint. One rule is strict: responses with status 429 must not be stored by a cache.
On the client side, the right behaviour is to honour Retry-After when present, and otherwise back off exponentially with random jitter. Hundreds of clients retrying in the same second simply recreate the load the limit was protecting against. Retried writes should carry idempotency keys so they are not applied twice.
What Googlebot does with a 429
Google singles 429 out from the other 4xx codes: it reads it as a signal that the server is overloaded and treats it as a server error. Crawlers temporarily slow down, and once the server returns 2xx responses again the crawl rate climbs back gradually. Google's documentation does not say that Googlebot honours the Retry-After value, so a long delay in that header gives you no predictable control over crawling.
If you urgently need to reduce Googlebot crawling, Google recommends returning 500, 503 or 429, but for no more than a day or two. If the same URL keeps returning these codes for several days it may be dropped from the index, and prolonged use can make Google slow down or stop crawling the site for good. On large sites that translates directly into lost crawl budget.
429s aimed at the wrong clients
Most sites don't produce 429s on purpose; a rate rule in the CDN or security layer does. A crawler fetching many pages in a short burst can look just like a malicious bot. Tune rules to real user behaviour and handle verified search engine crawlers separately, identifying them by reverse DNS or published IP ranges rather than by user agent, which is trivial to fake.
The line between 429 and 503 matters too: 429 says one particular client exceeded its limit, while 503 says the service is temporarily unavailable to everyone. Planned maintenance calls for 503, not 429.