Jev 429 Error: Rate Limits and Retry Fixes

What 429 means

You exceeded a provider limit. TypeSafe currently lists 250000 tokens/second and 1200 requests/minute for the direct API. These limits can change without notice while capacity adjusts. A provider gateway can impose different limits.

Tokens and requests are different budgets

Many tiny requests can exhaust RPM. A burst of large states can exhaust tokens/second. Average daily usage does not describe a burst. Measure concurrent requests, estimated input size, and the provider’s returned usage together.

Honor Retry-After

If the response includes Retry-After, use it. It may be seconds or an HTTP date. Bound the total job deadline so a long wait becomes a queued retry or review item instead of tying up a request indefinitely. Official SDKs handle retry headers and backoff by default; check your installed version’s policy.

Exponential backoff for raw HTTP

The following helper calculates a bounded delay and honors both header forms. The caller should stop after a small number of attempts and apply an overall deadline.

import random
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

def retry_delay(attempt, retry_after=None):
    if retry_after:
        try:
            return max(0, float(retry_after))
        except ValueError:
            try:
                return max(0, (parsedate_to_datetime(retry_after) -
                               datetime.now(timezone.utc)).total_seconds())
            except (TypeError, ValueError):
                pass
    return min(30, 2 ** attempt) + random.uniform(0, 1)

Do not wrap a retrying SDK in another unbounded retry loop. For response codes 401 and 422, fix the credentials or schema first. A 529 overload can use a transient-failure policy similar to 429.

Reduce bursts

Use a bounded queue and cap concurrency. Combine independent questions to avoid repeated state. Smooth scheduled jobs across time, and apply per-tenant quotas so one customer does not consume the shared budget.

Recovery and observability

Record attempts, retry reason, cumulative wait, and final outcome without logging secrets. Alert on a sustained increase in rate-limit failures. For consequential downstream operations, use idempotency and a durable job record so successful evaluation is not confused with successful execution.