validator_failures

What this detector catches

Any call to mesedi.validator_result(name, passed=False, ...) records a validator_result event and, on execution close, groups the affected executions under failure_class = validator_failures with the validator name as the signature. The optional severity argument is honored end-to-end (since the SDK severity-honoring fix shipped): pass severity="warning" to route the failure group through your warning-tier webhooks instead of the default critical-tier.

Validator failure messages pass through the DLP scanner before storage — PII patterns in your message or reason fields are redacted at the trust boundary, same as LLM prompts.

Example failures it groups

• output_schema_match — the model returned malformed JSON your schema check rejected.
• pii_redaction — PII slipped past your redaction pass.
• length_within_bounds — the model wrote too much or too little.
• factuality_check — a downstream factuality evaluator flagged a hallucination.
• Any custom validator your team wrote and wired into the agent's output path.

What to do when it fires

Open the failure-group detail page. The Playbook walks through two diagnostics: read the validator_result event payload (the rejection reason, if your validator emits one, tells you which field failed + what the expected vs actual values were), and read the llm_call event immediately beforeit (the failing output is in the assistant message). Then resolve to one of three fixes: the model is wrong (tighten the prompt with an example of valid output OR use function-calling / structured-output), the validator is wrong (loosen if over-strict — false-positives desensitize you), or the prompt is wrong (re-derive the prompt from the validator's success criterion to close the gap). The playbook also includes a useful batch-debugging shortcut for high-count groups: read 10 failed outputs together and look for the structural failure pattern.

When NOT to investigate

• Validators that legitimately fail by design — an is_disposable_email check that intentionally fails for known throwaway addresses, a content-moderation validator surfacing flagged inputs you intend to reject. Allowlist the validator name via Allowlist so matching failures skip grouping + skip webhook delivery.
• Informational-tier checks you want surfaced but not paged on. Pass severity="info" to the SDK call; the failure group is created but routed only to webhooks configured for the info severity tier (set on the Severity routing surface in Settings).
• First-fire after a new validator deploys. A spike on a freshly-added validator is usually discovery, not regression — the new check is finding pre-existing bad outputs the old code was letting through.

Tunable thresholds + allowlist

No per-project tunable thresholds — every passed=false validator fires the detector. Use the Allowlist primitive to suppress legitimate-fail validators by name. Severity routing (configured in Settings) lets you downgrade specific validator failure groups to warning or info tier without suppressing them entirely.

Related docs

• tool_failures — for failures coming from external dependencies; validator_failures is for failures decided by your own code.
• grounding_failure — for evaluator-driven groundedness scores; uses a different SDK helper (emit_eval_score) but covers an adjacent failure-detection problem.
• Allowlist — suppress legitimate-fail validators.