Docs / Observability / Record integrity

record_integrity

Every other detector in Mesedi asks whether your agent did something wrong. This one asks a different question: is the record of what your agent did internally consistent? It fires when an execution’s event stream is missing a sequence number, or when two events claim the same one.

Signatures

• record_integrity:sequence_gap — at least one sequence number is missing between the lowest and highest the execution recorded. Events were produced that this record does not contain.
• record_integrity:duplicate_sequence — two or more events claim the same sequence number, so one position in the record was written more than once.

Both can fire on the same execution. Ordering is deterministic: gap before duplicate.

This is not a tampering alert

A missing sequence number is far more often a dropped request, an SDK process killed before its buffer flushed, or a retry that landed twice than it is anyone removing anything. Treating this as a security event will waste your time in almost every case.

There is a harder reason too, and it is structural rather than statistical: the detector’s only input is supplied by the caller. Sequence numbers arrive inside the event body, and the ingest endpoint authenticates your project key, not the content of what is posted. Anyone able to post events chooses the sequence numbers — so an actor concealing something does not leave a gap. They post a dense stream, and this detector reports a clean record.

The scope is therefore: it sees records that were damaged, not records that were authored. Loss, crash and retry it catches. Fabrication it does not, and no detector reading the same unsigned stream could — that would be asking the data whether the data is trustworthy.

What it establishes is narrower and still worth having: this record is not complete, and here is the exact position that is missing or doubled. Whether that is infrastructure or intent is a question this data cannot answer. Proving a record was not altered after the fact requires it to have been signed and chained at the moment it was written — a different mechanism than this one.

What it cannot see

The gap check measures from the lowest sequence number actually present, not from a fixed starting number, because Mesedi must not assume where your SDK begins counting. The consequence is that if the first events of a run never arrived, the lowest survivor silently becomes the new floor and the record looks clean. Detecting a truncated head needs a start marker the event stream alone does not carry.

Why it is clock-independent

The two shipped signals are pure integer analysis on sequence numbers. No timestamp is consulted, so nothing here can be perturbed by clock skew between hosts, an NTP correction, or a container drifting. Two further signals were designed and deliberately cut for exactly that reason — timestamp regression and events falling outside their execution’s own window are both real, both clock-dependent, and both would fire constantly on any customer running agents across more than one machine unless a per-project skew tolerance shipped first.

Common causes

• Transport loss. An event POST failed and was never retried. The most common explanation and the easiest to confirm from your SDK error logs.
• Process death mid-flush. OOM, container eviction, deploy rollout or timeout kill while events were buffered. A gap at the end of the range usually means this.
• Duplicate delivery. A retry succeeded after the original also succeeded. Produces a duplicate without a gap.
• Concurrent writers sharing a sequence counter. Several workers emitting under one execution id without coordinating. Usually produces duplicates at several positions.

No configuration

This detector is correctness-only. There is no threshold to tune, because a missing sequence number is either missing or it is not. It also requires no new instrumentation: it reads sequence numbers every SDK version has always emitted, so it works on telemetry you are already collecting.

Related: crashes (an execution that ended badly often also lost its tail) and provider_incident (a provider outage covering the same window often explains the loss).