Mesedi
Start free →

Docs / Observability / Drift

drift

Something upstream of the agent changed without coordination. Two detectors share the drift failure_class because they catch the same operational pattern from different angles: categorical drift (a new model identifier showed up) and distributional drift (the prompts shifted continuously). Either is often the leading indicator that a prompt change, retrieval-source update, or model swap quietly changed output quality before any louder detector caught it.

What this detector catches

At execution close the backend runs two independent checks. Either or both can fire.

Model drift (categorical) — the execution used a model identifier the project hasn't seen in its recent history. Fires under signature new_model:<model_name> (alphabetically- first new model when several appear simultaneously). Catches uncoordinated model swaps, misrouted A/B variants, and stale-config-pointing-at-the-wrong-model bugs. Day-one suppression is built in: if the project has no historical models, drift is not fired (every model would otherwise look new).

Lexical drift (distributional) — the execution's llm_call user_messages diverged from the project's 3-gram baseline beyond a configured cosine cutoff. Three severity buckets:

  • lexical_drift_0.45+ — cosine distance ≥ lexical_threshold_low (default 0.45). Mild shift; usually noise but worth tracking.
  • lexical_drift_0.55+ — distance ≥ lexical_threshold_medium (default 0.55). Real shift; usually correlated with something that shipped.
  • lexical_drift_0.70+ — distance ≥ lexical_threshold_high (default 0.70). Significant shift; investigate immediately.

Both signature families live under the drift failure_class. The dashboard renders them as distinct rows so the cause is visible at a glance: a new_model:* row is a routing / config investigation; a lexical_drift_*+ row is a prompt-pipeline investigation.

What it does NOT catch

Semantic-but-lexically-similar drift — if your prompts shift from "what does this policy require" to "what is the policy's requirement," the two sentences share nearly-identical 3-gram bags so lexical drift won't fire. Embedding- based semantic drift is not currently shipped; if you suspect this kind of drift, run your own embedding pipeline against execution prompts as a complement.

Provider-side weight refreshes behind the same model alias — if Anthropic refreshes claude-3-5-sonnet without a new identifier, OpenAI tweaks gpt-4o weights, or Vertex auto-upgrades gemini-flash, the model-drift detector cannot fire (the identifier didn't change). The lexical detector may catch the downstream output shift, but the cause attribution lives in the provider changelog, not in Mesedi. Treat drift firing alongside a provider rollout window as a strong hint.

Path-stable drift attribution — the detector doesn't consider WHICH agent or code path used a new model. For projects running heterogeneous agents, false positives are possible when a new agent legitimately introduces a new model.

Drift is also a heuristic, not ground truth. Small lexical shifts in long-tail agent traffic can fire even when output quality is fine. The three-bucket design is intentional: low surfaces noise, high surfaces signal.

What to do when it fires

Open the failure-group detail page. The Playbook walks through the "what changed?" diagnostic: read what shipped recently (prompt changes, retrieval source updates, tool schema changes, model swaps); compare affected execution outputs against a recent baseline execution to see HOW the lexical pattern shifted; if you're also seeing hitl_rejection_spike firing on the same window, human reviewers are independently noticing the same regression. If you're ALSO seeing provider_incident, the root cause is probably provider-side.

When NOT to investigate

  • Low-bucket noise — lexical_drift_low fires at the noise floor by design. Treat low-bucket clusters as informational; wait for medium or high before pulling the alarm.
  • Intentional prompt experiments — if you just shipped a new prompt, drift firing is the expected signal that the new prompt produces different outputs. Validate the new outputs are better; the cluster is a feature, not a bug.

Tunable thresholds

All three cutoffs are per-project tunable via the Detector thresholds primitive: lexical_threshold_low (default 0.45), lexical_threshold_medium (default 0.55), lexical_threshold_high (default 0.70). Cross-threshold ordering ( low < medium < high) enforced detector-side; violation reverts to defaults. No tier cap (alerting sensitivity). Customer-tuned signatures embed the cutoff value so dashboard filters stay meaningful for tuned projects.

Related docs