grounding_failure

What this detector catches

Customer code calls mesedi.emit_eval_score(evaluator, metric, score, passed=...) after running an evaluation. The detector fires under two priorities:

• Explicit pass=false — if the evaluator reported the output failed (regardless of score), fires under grounding_failure:<evaluator>:<metric>. Customer verdict outranks heuristic. This pass runs first.
• Mean below the floor — when explicit pass=false didn't fire, the detector computes the mean of THIS execution's higher_is_better=true eval_score events per (evaluator, metric) and fires when the mean drops below the floor (default 0.5 via mean_floor). Inverse metrics like hallucination_rate are excluded from the mean rollup; they need different threshold semantics.

The detector emits all distinct (evaluator, metric) failures per execution, capped at 20. A RAG pipeline whose Ragas faithfulness, Ragas answer-relevancy, AND HHEM hallucination all flag the same output produces three failure_groups, not one. Same (evaluator, metric) pair triggering both pass=false and mean-floor fires once (dedup prevents double-emit).

Supported evaluators (helper SDK)

Mesedi ships first-class helpers for 3 popular evaluators so you don't have to write the glue:

• Ragas — the open-source RAG evaluator. Call mesedi.emit_ragas(result) after your Ragas run to forward all metrics + scores in one call.
• Promptfoo — general-purpose LLM evals. mesedi.emit_promptfoo(test_result) wires test outcomes directly.
• Vectara HHEM — hallucination evaluation. mesedi.emit_hhem(result) takes the HHEM output directly.
• Any custom judge — the underlying emit_eval_score(evaluator, metric, score, passed=...) is fully generic; the helpers above are conveniences.

What to do when it fires

Open the failure-group detail page. The Playbook walks through the diagnostic: identify the evaluator + metric from the signature; read the failing outputs alongside the evaluator's reason field if it emits one; decide between fixing the prompt (the model is wrong), fixing the evaluator (the eval is over-strict), or fixing the retrieval (the context is missing for RAG groundedness failures). The detector framing applies equally to any evaluator-driven failure — RAG groundedness, factuality scoring, safety filters, custom domain checks.

When NOT to investigate

• New evaluator + small sample — a freshly-wired evaluator firing on the first few interactions is discovery, not regression. Wait for 20+ interactions before treating as actionable.
• Evaluator-itself is the problem — if the same evaluator + metric pair is firing on outputs you manually verify as correct, the evaluator is over-strict. Tune it or replace it; this cluster is informational not actionable.

Tunable thresholds

Two per-project knobs via the Detector thresholds primitive (neither tier-capped — alerting sensitivity only, no cost asymmetry):

• mean_floor (default 0.5; bounds [0.0, 1.0]). The global default floor applied to any (evaluator, metric) pair not in the per-evaluator override map.
• per_evaluator_floors (default empty map; max 50 entries; values bounded [0.0, 1.0]). Per-evaluator override keyed "evaluator_id:metric_type" → float. Lets one strict evaluator have a tighter floor without disturbing the rest. Lookups fall back to mean_floor when a key is absent.

The 20-match-per-execution cap is hardcoded. No allowlist support today.

Related docs

• validator_failures — the boolean-pass-fail sibling. validator_failures is for code-driven yes/no checks; grounding_failure is for score-driven evaluator outputs.
• hitl_rejection_spike — the human-judge counterpart to the evaluator-judge signal.