infrastructure_throttled

What this detector catches

An infrastructure_event with reason indicating throttling (rate_limit, circuit_breaker, or quota_exhausted) groups under one of:

• infrastructure_throttled:rate_limit:<provider> (or :rate_limit:<provider>:<dimension> when the event names a specific axis like tokens_per_minute)
• infrastructure_throttled:circuit_breaker:<provider>:<state> (state defaults to open)
• infrastructure_throttled:quota_exhausted:<provider>

The optional retry_after_ms field on the event payload is a separate signal — providers report it in Retry-After headers and the instrument_*modules capture it. It's a remediation hint, not a reason value, so it does not appear in the signature.

The detector runs after tool_failures in the chain so throttling wins over the symptomatic tool failure (the right priority — the underlying cause is the throttle, not the tool that failed because of it).

Auto-emit from instrument_* modules — Anthropic, OpenAI, Cohere, and Gemini instrumentation wrappers all auto-emit infrastructure_event on canonical rate_limit + quota_exhausted exceptions (sync + async + streaming paths). You get the signal for free without writing retry-wrapper code. The manual emit_infrastructure_event SDK helper is still available for custom paths (non-LLM HTTP calls, your own circuit-breaker layer, queue-side throttling).

Example failures it groups

• Anthropic API returns 429 with retry-after: 30. instrument_anthropic auto-emits an infrastructure_event; the detector fires under infrastructure_throttled:rate_limit:anthropic (or with a :tokens_per_minute suffix when the dimension is named).
• OpenAI returns 429 with quota-exhausted message. Auto- emit fires infrastructure_throttled:quota_exhausted:openai.
• Your own circuit breaker around a flaky internal service trips and you call emit_infrastructure_event(reason="circuit_breaker", provider="my-service"). Fires infrastructure_throttled:circuit_breaker:my-service:open.

What to do when it fires

Open the failure-group detail page. The Playbook walks through the operational fix tree: identify the throttling source (which provider, which endpoint, what tier); decide between raising the rate limit (most expensive, instant), exponential backoff with jitter (cheapest, delayed), or switching to a fallback provider (most resilient, requires implementation). Read the retry-after value from the event payload — providers usually tell you exactly how long to wait, and your client should honor it.

Relationship to provider_incident

infrastructure_throttled is the per-tenant signal — your project hit a throttle. provider_incident is the cross-tenant signal — many projects hit the same provider issue at the same time. If your infrastructure_throttled group is firing and you ALSO see a recent provider_incident for the same provider, the root cause is provider-side (their fault, not yours). If only infrastructure_throttled is firing, the root cause is per-tenant (your traffic shape, your quota tier, your retry pattern). The two detectors are designed to read together.

When NOT to investigate

• Expected burst traffic — if your workload has known peaks (campaign launches, batch jobs), the detector firing during those windows is informational. Use severity routing to downgrade.
• Backoff is working — if your retry layer is honoring retry-after and eventually succeeding, the detector is signaling the contention not a real failure. Same severity-routing pattern.

Tunable thresholds + allowlist

No per-project tunable thresholds today. No allowlist support. Use severity routing for expected-throttling patterns you'd rather monitor than alert on.

Related docs

• context_overflow — operating-environment sibling: tokens crossed the window. infrastructure_throttled is the rate-side signal; context_overflow is the size-side signal.
• provider_incident — the cross-tenant version of the same root cause. Read together.