Docs / Observability / Cascading failure

cascading_failure

A multi-agent handoff was followed by the child sub-agent crashing or timing out. Mesedi clusters the parent + child into one failure_group so you see one incident view (not two separate alerts for the same root cause). The detector joins across executions via parent_execution_id and surfaces the first-failing-edge signature: the agent_handoff event from parent to child plus the failure terminal status of the child.

What this detector catches

When a parent execution emits an agent_handoff event (via the emit_agent_handoff SDK helper OR via one of the supported framework integrations: LangGraph, OpenAI Agents) and the child execution terminates in one of three failure terminal statuses — crashed, timeout, or validation_failed — the backend joins the two executions into a cascading_failure group. The signature shape is cascading_failure:<from_agent>:<to_agent>:<child_status> (from the handoff edge plus the child's terminal status). First-failing-edge wins by sequence order — the first handoff in the parent's timeline whose child terminated in a failure status takes the signature, so repeated cascades along the same edge cluster as one group across executions. halted child status is explicitly excluded (operator-driven, not a real failure); completed children are benign handoffs and never participate.

Example failures it groups

  • A planner agent hands off to a research sub-agent, which crashes parsing the LLM's output as JSON. Without cascading_failure, you'd see two separate signals: a crashes group on the child execution + an ignored handoff event on the parent. The cluster joins them into one "planner → research → JSON-parse crash" pattern.
  • A supervisor agent hands off to a tool-using sub-agent that hits a hard time_budget overrun. The cascading group surfaces the chain so you can decide whether to raise the child's budget vs change the supervisor's handoff strategy.
  • A multi-step research pipeline where step N's output crashes step N+1's parser. The cluster groups all (step_N → step_N+1) failures regardless of the underlying input, so structural pipeline bugs surface immediately.

What to do when it fires

Open the failure-group detail page. The Playbook walks through the "two views of the same incident" framing: the parent execution shows the handoff context + the inputs it gave the child; the child execution shows what failed and how. Read both timelines side by side. Common remediation patterns: tighten the schema the parent feeds to the child (the most common cause is the parent producing output the child can't consume), bound the child's budget so the failure is detected earlier, or restructure the handoff so the parent doesn't depend on the child's success for its own completion. If multiple parents are failing into one child, the child agent itself is brittle — treat that as a single-agent debugging session against the child execution.

When NOT to investigate

  • Operator-halted executions — the detector explicitly excludes haltedchild status (you killed it on purpose; it's not a real failure). If you see halted-child cascades, your halt path is being misclassified upstream.
  • Fire-and-forget spawn handoffs — if your topology uses handoff_kind = "spawn"and the parent's success genuinely doesn't depend on the spawn-child's outcome, a spawn-child failure is closer to a supervision gap than a cascade. Turn on the per-project exclude_spawn_handoffs knob to skip spawn rows from cascade scoring.
  • Long-gap cascades — if a child terminates hours after the parent handed off (long-lived spawn workers, queued background jobs) and grouping the two as a cascade is misleading, tighten cascade_window_seconds(e.g. 300 for a 5-minute join window). Default 86400 (24 h) preserves the historical "no window" behavior.

Tunable thresholds + allowlist

Two per-project knobs via the Detector thresholds primitive (neither tier-capped):

  • cascade_window_seconds (default 86400 = 24 h; bounds [10, 86400]). Rows where child_ended_at - handoff_emitted_at exceeds this are excluded from scoring. Tighten when long-gap spawn children are showing up as false- positive cascades.
  • exclude_spawn_handoffs (default false; boolean). When true, skip rows where handoff_kind = "spawn" before scoring.

Both defaults preserve the original detector behavior. The allowlist primitive does not currently support this detector; use severity routing if you need quieter alerting beyond what the two knobs above allow.

Related docs

  • coordination_deadlock — the multi-agent sibling that catches circular handoff patterns (A→B→A or longer cycles) where each agent is waiting on the other to release control. cascading_failure catches linear failure chains; coordination_deadlock catches cyclic stalls.
  • Multi-agent topology + handoffs — the overview doc covering parent/child execution graph + the agent_handoff event + how the multi-agent detectors cluster cross-execution signals.