Docs / Observability / Tool schema drift

tool_schema_drift

A third-party API silently changed its return shape. Your agent is now interpreting an unfamiliar schema and probably getting the wrong answer — but no exception was raised, no validator fired, the execution looks successful. Mesedi fingerprints tool return values across a baseline period and surfaces when a new shape appears that doesn't match any baseline pattern.

What this detector catches

For every tool_call with status=success, the backend computes a structural fingerprint of the return value (sorted keys + value type names; values themselves are dropped). The historical roll-up tracks the dominant shape per tool. Drift fires when three conditions hold: (1) the tool has at least min_history_callsprior successes (default 10), (2) a single shape covers at least 2/3 of that history (stable baseline), and (3) the current execution's shape differs from that baseline. Fires under tool_schema_drift:<tool_name>:<hex8> where the hex8 is the first 8 chars of SHA-256 over the shape. Tools without a stable baseline (heterogeneous shapes) never fire drift; tools in priming (fewer than min_history_calls successes) never fire either.

What the shape walk preserves and drops

Primitives become their type name (null, bool, string, number). Integers and floats both render as number— the shape doesn't split numeric subtypes, so a tool returning {price: 12} and {price: 12.5} clusters identically.
Arrays render as [<element-shape>]using ONLY the first element. Length is dropped; that's the desired property. Heterogeneous arrays are hashed deterministically but a tool returning [int, string] then [string, int] would NOT cluster together because the first-element shape differs.
Objects sort keys alphabetically and emit {key:shape,...}. ALL keys participate — there's no optional-field tolerance. A field that appears in some responses but not others produces a different shape (and may legitimately trip drift if the absence is structural).
Typed sentinels — the SDK ships non-JSON-native values (datetimes, custom Python/JS objects) as {"__type__": "datetime", "value": ...} or {"__type__": "object", "class": "User", ...}. When an object carries a string __type__ marker, the shape emits <typed:datetime> or <typed:object:User> instead of collapsing to a generic{__type__:string, value:string} shape. Without this, every typed sentinel would mask the drift signal.

Return-value byte cap + truncation telemetry

Tool returns can be large (multi-MB document fetches, full HTML pages, long search results). Mesedi applies a per-project byte cap at the handler layer BEFORE the drift detector sees the data — returns above the threshold are excluded from fingerprinting (treated as inconclusive rather than as a new shape). Default 8 KB; tunable via tool_return_value_max_bytes (tier-capped: consult Settings for your current limit). The Settings dashboard surfaces truncation-rate telemetry: if your byte cap is firing frequently, the detector is silently excluding those tool calls from baseline + comparison. Raise the cap if the telemetry shows a high truncation rate AND the excluded tools matter for drift coverage.

What to do when it fires

Open the failure-group detail page. The Playbook walks through the diagnostic: read the new fingerprint alongside one of the baseline fingerprints; identify what changed (key added, key removed, type flipped, array-vs-object mismatch); decide whether the API vendor changed their contract or whether your tool wrapper is producing different shapes for different inputs (e.g. authenticated vs unauthenticated response). Common remediation: update your tool wrapper + parser; pin the API version if the vendor supports it; file a bug with the vendor if the breaking change was undocumented.

When NOT to investigate

Tool with intentional polymorphic returns — if your tool wrapper deliberately returns different shapes for different inputs (e.g. authenticated vs anonymous, different account tiers), the detector will fire on every shape variant. Tune the baseline calls up or use severity routing.
Priming-stage signals — the detector is conservative in priming. If you're seeing "priming N/10 calls" signals, that's the detector building baseline, not an alert.

Tunable thresholds

min_history_calls (default 10; bounds [2, 1000]) is per-project tunable via Detector thresholds. tool_return_value_max_bytes (default 8192; bounds [1, 1M]) is also per-project tunable — see the Settings tile for the truncation-rate telemetry that helps you decide whether to raise it.

Related docs

tool_failures — for tools that raise exceptions. tool_schema_drift is for tools that return new shapes silently.
drift — the LLM-output-side counterpart. tool_schema_drift is about API contracts; drift is about model outputs.
Detector thresholds