Docs / Observability / HITL rejection spike
hitl_rejection_spike
An unusually high fraction of recent human-in-the-loop interactions came back rejected or editedby the reviewer. This is one of the most informative signals in Mesedi's detector set because rejection is a high-confidence labeled signal: a human looked at the output and decided it was wrong (or wrong enough to require editing). The detector uses basis-points integer arithmetic (no floating-point rounding) and a defensive minimum-sample gate to prevent firing on noisy small samples.
What this detector catches
At execution close, the backend examines recent HITL interactions for the project. When the rejection rate exceeds 40% over a minimum of 5 interactions, the detector fires with hitl_rejection_spike:rejected signature. When the rejection rate is healthy but the edit rate exceeds 30% over a minimum of 5 interactions, it fires with hitl_rejection_spike:edited signature. Rejected-wins-over-edited is the correct priority — outright rejection is a stronger signal than edit. The minimum-sample gate of 5 is intentional: 1 rejection out of 3 interactions would otherwise fire at 33% rate which is meaningless statistically.
Example failures it groups
- A content-generation agent's outputs are being rejected more than 40% of the time after a prompt change. The signal usually surfaces within 10-20 interactions of a regression.
- An auto-coding agent's diffs are being edited by reviewers more than 30% of the time (still accepted, but with substantial human touch-up). Quality drift signal.
- After a model swap, the rejection rate on customer-service auto-responses jumped from 5% to 45%. The model swap is almost certainly the cause.
What to do when it fires
Open the failure-group detail page. The Playbook walks through the "agent quality probably regressed; what changed?" diagnostic: check what shipped recently (model swaps, prompt changes, retrieval source updates, tool schema changes). The Playbook also cross-links to other Mesedi detectors that explain quality regressions from different angles: drift (output distribution shift), tool_schema_drift (tool contract changes), and provider_incident (provider-side model degradation). Reading these alongside the rejection-spike group usually identifies the proximate cause within minutes.
When NOT to investigate
- New reviewer onboarded — a fresh reviewer who's stricter than the team baseline can push the rejection rate up while agent quality is unchanged. Wait 1-2 weeks for calibration.
- Sample size borderline — if you're seeing the alert at exactly 5 interactions, treat it as informational and wait for 10+ interactions before treating it as actionable. The minimum-sample gate is statistical guardrail, not ground truth.
- Intentional high-edit pattern — if your product design is "agent drafts, human polishes," the edit rate is your product metric, not a failure signal. Use severity routing to downgrade the failure class to info-tier.
Tunable thresholds + allowlist
The 40% rejection rate / 30% edit rate / minimum-sample-5 thresholds are hardcoded statistical guardrails — not customer policy. Per-project tunability for these knobs is not currently exposed. The measurement window is also currently fixed. The allowlist primitive does not currently support this detector.
Related docs
- hitl_timeout — the HITL sibling that catches non-responding humans (timeouts + SLA exceedances). hitl_rejection_spike catches disagreeing humans.
- drift — upstream cause: output-distribution shift.
- tool_schema_drift — upstream cause: tool contract changes.
- provider_incident — upstream cause: provider-side regression.
- HITL SDK overview — the helper API.