429 too many requests: fixes for web scraping and AI APIs

Imagine a scraper collecting public product pages through CyberYozh App proxies, then sending the extracted text to an AI API for analysis. Partway through the job, it stops with “429 too many requests.” The tempting response is to retry, but first you need to find out where the request failed.

Did the website, an intermediary service, or the AI API return the error? Each can enforce different limits, so changing the wrong setting may leave the problem unresolved.

During collection, check request pacing, concurrent connections, and whether the limit applies to an IP or session. CyberYozh App lets you control proxy IP changes and session behavior at this stage. During AI processing, check the provider’s request and token allowances instead. This guide walks through both stages so you can identify the restriction, apply the right fix, and avoid repeating work that already succeeded.

TL;DR

• 429 too many requests means a service is enforcing a rate or usage limit. It does not automatically mean your proxy is defective or permanently banned.

• Find the source first: the target website, an intermediary service, or the AI API. Inspect the response body and headers.

• Slow down and retry carefully. Respect Retry-After, coordinate workers, reduce concurrency, and use backoff with jitter and a maximum attempt count.

• Save completed work. Retry the failed collection or analysis step without restarting the entire workflow.

• Check AI request limits, token allowances, and quotas separately. Changing IPs does not increase account allowances.

• Match proxy infrastructure to collection requirements. CyberYozh App provides geotargeting, sticky sessions up to 24 hours and unlimited traffic.

What does 429 too many requests mean?

A 429 too many requests error means a service is limiting further requests because you have reached one of its usage thresholds. Usually, this involves sending too many requests within a particular time window. The response alone does not tell you which limit was reached or how long the restriction will last.

To understand what “429 too many requests” means for your workflow, check how the service counts activity. As MDN’s 429 documentation explains, limits may apply across a server, to a particular resource, or according to an IP address, authenticated user, or application. Several workers can therefore share an allowance even when each sends relatively few requests.

A 429 does not, by itself, establish a permanent ban or prove that your proxy is defective. The connection may be working while the destination refuses additional requests. Changing IPs will not reset a limit tied to the same authenticated account.

Inspect the response body and headers for details about the restriction, including any Retry-After instruction, before choosing your next action.

Find out which service is returning the error

Return to the product-page example: did collection fail before the text reached the AI API, or did analysis stop after the pages were saved?

In a browser, open developer tools, select the Network panel, and inspect the request with status 429. In Postman, review the response status, headers, and body. In application logs, match the failure to its timestamp and workflow step.

Record the hostname, response body, relevant headers, timestamp, and request ID when available. Remove passwords, API keys, authorization values, and session cookies before sharing diagnostic records.

How to fix error 429 too many requests

Once you identify the service returning 429, let the response details guide the fix. Work through these steps before restarting the full job.

1. Pause repeated attempts
   Pause the affected job, then check background workers, scheduled tasks, and duplicate runs. If they share the same allowance, coordinate their pause too; stopping one script may leave most of the traffic unchanged.

2. Respect the Retry-After header
   Retry-After: 60 means wait at least 60 seconds. An HTTP date specifies when to try again. MDN’s Retry-After documentation explains both formats. When the header is absent, consult the service’s documented limits and use increasing retry delays rather than assuming a universal waiting period.

3. Reduce request frequency and concurrency
   Frequency means requests sent over time; concurrency means requests running simultaneously. Control both. Workers sharing an allowance need coordinated pacing, such as a shared queue with a rate limiter, because individual delays do not control their combined traffic.

4. Use exponential backoff with jitter
   Increase delays after repeated failures and add jitter, a small random variation that helps workers avoid retrying together. Set a maximum attempt count and check the client or SDK’s existing retry behavior before adding another loop. Never shorten the server’s requested wait.

5. Remove unnecessary requests
   Deduplicate queued URLs, cache successful responses where permitted, and reduce polling when fresh data is unnecessary. Save completed pages so one failed request does not restart the entire collection job.

This Python GET example handles both header formats, falls back to backoff for missing or malformed values, and stops after five unsuccessful attempts:

import random
import time
from datetime import timezone
from email.utils import parsedate_to_datetime

import requests


def get_with_backoff(url):
    for attempt in range(5):
        response = requests.get(url, timeout=30)
        if response.status_code != 429:
            response.raise_for_status()
            return response
        if attempt == 4:
            response.raise_for_status()

        value = response.headers.get("Retry-After", "").strip()
        delay = 2 ** attempt
        try:
            if value.isascii() and value.isdigit():
                delay = max(delay, int(value))
            elif value:
                retry_at = parsedate_to_datetime(value)
                if retry_at.tzinfo is None:
                    retry_at = retry_at.replace(tzinfo=timezone.utc)
                delay = max(delay, retry_at.timestamp() - time.time())
        except (ValueError, TypeError, OverflowError):
            pass  # Invalid header: keep the backoff delay.

        response.close()
        time.sleep(delay + random.uniform(0, 1))

Fix recurring 429 errors in web scraping

Recurring 429 errors call for a review of the collection job’s overall schedule. Count requests across every worker targeting the same website, especially when they share an IP address, account, or API credential. Coordinate workers sharing a limit through a common rate limiter, and apply cooldowns across the affected group when throttling occurs.

Browser automation can generate more traffic than your script suggests. Opening one product page may trigger requests for images, scripts, recommendations, and background updates. Inspect the browser’s Network panel to see what actually reaches the target. Skip unnecessary resources only when doing so preserves the page behavior and data you need.

Deduplicate queued URLs while preserving intentional differences, such as regional storefronts or product variants. Save successful responses and record which items have finished. In the opening product-page example, if collection stops halfway through the catalog, resume from the failed item after the required wait. The pages already collected should remain available for AI processing without being downloaded again.

Preserve session continuity when related requests depend on the same cookies, account, or IP address. Base proxy rotation on the task’s connection requirements and the identified limit; a 429 alone is not a reason to change addresses. Schedule collection around the target’s documented allowances, and reduce polling when product information does not need frequent updates.

Fix 429 errors in AI APIs and agent workflows

Your scraper may collect every product page successfully, then hit a 429 when it sends the results for analysis. Finishing a collection batch can trigger a burst of model calls, each consuming part of the AI provider’s request or token allowance.

Check which allowance you reached. Claude’s rate-limit documentation distinguishes requests per minute, input tokens per minute, and output tokens per minute. Tokens are the units the model uses to process text. A few large inputs can therefore reach a token limit even when the number of requests looks modest. Check the limits applicable to your account and model.

Queue the analysis jobs and release them at a controlled pace. If several agents share an organization or project allowance, coordinate their combined usage. Reducing unnecessary input and requesting concise output can also reduce token consumption, while pacing prevents completed scraping batches from overwhelming the processing stage.

A log entry such as last status: 429 too many requests records the final HTTP status; diagnosis still depends on the underlying response. Inspect the provider’s error type, message, request ID, and limit headers, then compare them with usage and billing information. A temporary rate limit and an exhausted allowance require different actions. Claude, for example, documents a monthly tier spend-cap response that continues failing until access resumes.

Keep the collected pages saved independently of the analysis job. When an AI request fails, retry only that processing step after addressing the restriction. Check the SDK’s existing retry behavior so an outer workflow does not repeatedly restart calls that the client is already retrying.

If the response conflicts with observed usage, check the provider’s service-status page and investigate other workloads sharing the allowance. Retain the timestamp and request ID for support. Changing the scraper’s proxy IP will not increase the AI account’s request or token limits.

Where CyberYozh App fits into the scraping workflow

Once diagnosis identifies a need for regional access, session continuity, or better control over collection connections, CyberYozh App provides several infrastructure options. Match the connection to the task, including how long related requests need to keep the same IP.

• Mobile proxies: Unlimited traffic on real LTE/4G/5G plans keeps bandwidth costs predictable for ongoing collection. Dedicated plans reserve the mobile channel for your workload, while shared plans use a channel alongside other customers. Dedicated connections offer manual and API-based IP changes, allowing you to schedule changes around task boundaries. 

• Rotating residential proxies: Use regional targeting for collecting location-dependent listings or prices. Choose rotation for independent collection tasks or sticky sessions up to 24 hours when related requests need temporary IP consistency. Traffic is billed by usage, with unused traffic rolling over to the next billing month.

• Static ISP proxies: A fixed address for the rental period suits recurring collection sessions that require the same IP. This provides continuity across separate runs without depending on a temporary sticky session.

• Dedicated datacenter proxies: These provide a fixed datacenter connection for workloads where the target accepts that network type. They can suit routine collection jobs that do not require a residential or mobile connection.

CyberYozh App supports proxy connections in Scrapy, Postman, Playwright, Puppeteer, and Selenium. Full API access lets scripts manage supported proxy operations alongside your existing collection code.