Docs / Observability / Tool failures
tool_failures
A tool_call event in the execution recorded status=failed: the tool raised an exception or timed out, and the agent caught it. The execution ran to completion producing potentially degraded output. Mesedi groups affected executions by the failing tool's name so the fix (which almost always lives in the tool's client code or its upstream dependency) is one cluster per tool.
What this detector catches
The SDK's @mesedi.tool decorator wraps tool functions and records a tool_call event for every invocation. When the tool raises or times out, the event records status=failed along with the exception type (timeout vs other). At execution close the backend groups failed tool calls under failure_class = tool_failures with the tool name as the signature.
MCP tool calls go through the same path — no separate detector needed. Soft-failures (the tool returned successfully but with a parseable-but-wrong value) are NOT caught today. Use a validator to catch soft-failures.
Example failures it groups
- An HTTP-based tool that raised
ConnectionErroron a transient network blip. - A search tool that hit its own rate limit and raised
429. - A database tool whose connection pool was exhausted.
- A tool whose timeout fired (the SDK records it as a tool failure with a timeout marker).
- Any
@mesedi.toolfunction that raised any exception the agent code subsequently caught.
What to do when it fires
Open the failure-group detail page. The Playbook walks through three diagnostics: read the tool_call event payload (tool name + arguments + failure type tell you transient vs permanent), inspect what the agent did NEXT (retry vs fallback vs give-up — each is a different bug shape), and compare 2-3 affected executions to localize structural bugs vs argument-construction bugs. Common remediation patterns: retry-with-backoff for transient failures; startup health-checks for permanent ones (credentials, schemas, permissions); tightened argument validation + corrective re-prompt for argument-construction failures; explicit degraded_response events when silent-degradation is intentional product behavior.
When NOT to investigate
- Known-flaky third-party tools you've already decided to monitor manually (rate-limited demo APIs, third-party search providers with known SLA gaps). Allowlist the tool name via Allowlist so the detector skips grouping for matching failures.
- Best-effort tools whose failure is expected (e.g. an
is_disposable_emailcheck that frequently fails for valid throwaway emails by design). Same allowlist remediation. - Provider-wide outage. If all of one provider's tools are failing at once, check the provider_incident group — it's the cross-tenant signal surfacing the same root cause.
Tunable thresholds + allowlist
No per-project tunable thresholds — every failed tool_call fires the detector. Use the Allowlist primitive to suppress known-flaky tools: add an entry whose allowlist_keymatches the tool name and matching failures skip grouping + skip webhook delivery while the entry's match-count increments.
Related docs
- crashes — for tool exceptions the agent did NOT catch (the exception escaped the wrap boundary).
- validator_failures — for catching the soft-failure pattern (tool returned a parseable-but-wrong value).
- provider_incident — for cross-tenant outages affecting many tools at once.
- Allowlist — suppress known-flaky tools.