provider_incident

What this detector catches

Errors from LLM provider clients are mapped to a canonical error class set (sourced from spec/error_classes.yaml): rate_limited, quota_exhausted, internal_error, service_unavailable, timeout, invalid_api_key, client_error, unknown. The detector clusters distinct tenants hitting the same (provider, error_class) within a sliding window and fires under provider_incident:<provider>:<error_class> when the count crosses min_tenants (default 2).

Only the provider-side subset of the canonical set triggers provider_incident clustering: rate_limited, quota_exhausted, internal_error, service_unavailable, timeout. The customer-side classes (invalid_api_key, client_error, unknown) are filtered out by IsProviderSideErrorClass — cross-tenant credential / request failures are usually project-wide rotation events or shared validation bugs, not provider outages, and surface elsewhere.

Auto-instrumentation across 4 providers

Mesedi's LLM instrumentation modules auto-map exceptions to the canonical error class and emit provider-incident-eligible events for free — no manual error-handling code required. Coverage:

• Anthropic — sync Messages.create, async AsyncMessages.create, Messages.stream + AsyncMessages.stream.
• OpenAI — sync + async Chat Completions + Responses API; streaming on both surfaces.
• Cohere — Client.chat + ClientV2.chat (sync + async) + chat_stream variants.
• Gemini — GenerativeModel.generate_content + generate_content_async (sync + async + streaming).

When the provider returns a retry-after header, the auto-emit pipeline extracts it onto the event so retry logic can honor it directly. A CI test guards against provider-exception-mapping staleness; new exception types added by providers are caught at build time.

What it does NOT cover

• Non-chat surfaces (embeddings, audio, image) — auto-emit covers the chat APIs across all 4 providers; non-chat surfaces are not covered today.
• Vertex AI Gemini — direct Gemini SDK is covered; the vertexai preview surface is not covered today.
• Other LLM providers (Mistral, AWS Bedrock, Azure OpenAI) — request additions for any provider whose error model can be cleanly mapped to the canonical set.

Relationship to infrastructure_throttled

infrastructure_throttled is the per-tenant signal — your project hit a throttle. provider_incident is the cross-tenant signal — many projects hit the same provider issue. If infrastructure_throttled is firing on your project AND provider_incident is firing for the same (provider, error_class), the root cause is provider-side. If only infrastructure_throttled is firing, the root cause is per-tenant (your traffic shape, your quota tier, your retry pattern).

What to do when it fires

Open the failure-group detail page. The Playbook walks through the cross-tenant verification: confirm the affected tenants are unrelated (different traffic patterns, different quota tiers); check the provider's status page; honor retry-after if the auto-extraction provided one; switch to a fallback provider if your stack supports it; surface the degradation to your callers honestly rather than silently retrying for hours.

Tunable thresholds

min_tenants (default 2) is per-project tunable via the Settings dashboard. Single-tenant customers typically set 1 (every provider error fires); multi-tenant SaaS customers may set 3-5 to require cross-tenant clustering before alerting. Tier-aware caps and server-side validation enforce realistic bounds.

Related docs

• infrastructure_throttled — per-tenant version of the same root cause. Read together.
• crashes — provider errors that the agent didn't catch surface as crashes; provider_incident surfaces the cross-tenant view of those same errors.
• hitl_rejection_spike — provider quality regressions can drive rejection spikes; this is one upstream cause to check when rejection_spike fires.