Duplicate logs: distinguish retransmission from repeated action
Before deduplicating logs, verify event identity, provenance and detection outcomes so that genuinely distinct actions remain visible and explainable.
Integration scope: Seculogik uses a source-agnostic architecture to ingest security alerts and events through its own data map. Wazuh and OpenSearch are the first integrations. Other named sources describe planned connector targets; their availability must be confirmed for each evaluation.
After collection resumes, several records appear to describe the same sign-in. Should an analyst count several attempts or just one? Removing similar records can make a dashboard easier to read while erasing genuinely separate actions. Keeping them without explanation can instead inflate an alert based on a threshold.
The useful question is not simply whether the records look alike. It is whether they represent the same occurrence under an identity contract whose guarantees are understood. This guide offers a tool-independent method for investigating duplicates, testing a correction and documenting its effect. It makes no claim about available Seculogik integration capabilities.
1. Separate the action, the event and its delivery
We recommend starting with three levels. An action is what happened in the system being examined. An event is a record a source produces about it. A delivery is an arrival of that record in your collection chain. Several deliveries can belong to one event; several events can also describe different aspects of one action.
A sign-in observed by an identity provider and by an application is not automatically a duplicate. The observations may contribute different information. Conversely, two arrivals of one source event do not establish two sign-ins. Correlating observations and removing a retransmission are different decisions.
Define the source, organisation, stream and affected period. Locate where the multiplication appears: source export, collector, transformation, storage or display query. Preserve a small set of retrievable references before changing processing. Do not export every log by default, and keep sensitive evidence in its authorised locations.
2. Check what the identifier actually guarantees
CloudEvents 1.0.2 defines event identity through the combination of source and id. A retransmission can retain that combination, allowing a consumer to treat it as a duplicate. This guarantee belongs to the CloudEvents contract and its implementation, not to every field labelled “identifier”. Source: CloudEvents specification
Our recommendation is to consult the source documentation and follow a sample through the entire chain. Who assigns the identifier? Does it identify an occurrence, an event type, a transaction or a record newly created on arrival? Is it preserved during recovery? Within what scope is it unique? An identifier assigned on every ingestion cannot, by itself, recognise a previous delivery.
OpenTelemetry also describes log.record.uid as an identifier distinguishing log records; its convention requires different values for distinguishable records. The convention is opt-in and marked as being in development in the documentation consulted. Do not assume every producer supplies it or fulfils that contract. Source: OpenTelemetry
When source identity is insufficiently documented, keep the conclusion “suspected duplicate”. A matching key proposed by your team remains a hypothesis to test. Check instance changes, counter resets and collisions across organisations before allowing records to be discarded.
3. Compare deliveries without confusing similarity with identity
Elastic ECS distinguishes event.id, event.hash and event.ingested: event identity, a fingerprint of raw content and arrival time in central storage. These fields serve different purposes. Their names do not remove the need to check how your pipeline populates them. Source: Elastic ECS event fields
For each candidate pair, we recommend comparing documented source identity, provenance, meaningful content, available timestamps and the collection route. A later arrival can be consistent with recovery without proving it. Look for collection restarts or authorised replays known to the team responsible for the stream.
An identical fingerprint establishes content equality under the chosen representation and algorithm, subject to their limitations; it does not automatically establish a unique action. Separate operations can produce identical text. Conversely, enrichment added on arrival can change a retransmitted record’s content. Document what the comparison includes and excludes.
In particular, avoid matching solely on an account, outcome and rounded timestamp. That key can merge several closely spaced attempts. If records sharing a documented identity contain conflicting meaningful content, preserve both and investigate the anomaly. Do not silently choose whichever arrived last.
4. Test deduplication before removing evidence
Our recommendation is to mark candidate duplicates first, using a test environment or parallel view. Compare results against the original view without erasing the sources needed for verification. Define the scope, how long deduplication state will be retained and the expected behaviour for late arrivals.
Prepare synthetic data containing no secrets and include at least these situations:
- One source event is delivered repeatedly with its identity preserved.
- Separate events have identical text and different identities.
- Different producers or organisations use the same local identifier.
- A delivery arrives outside the chosen matching window.
- An identifier is missing, or conflicting content shares one identity.
- An authorised replay passes through processing again.
Write the expected result for each situation before running the test. Check retained occurrences, references to grouped deliveries and anomaly handling. A lower displayed volume does not establish that the correction works. Confirm that a genuine repetition remains visible and that organisation boundaries are respected.
Entirely fictional example: two separate sign-in attempts share an account, outcome and displayed second, but have different source identities. Each is then delivered a second time. Under a verified identity contract, the deduplicated view should retain both occurrences with references to their retransmissions. It should neither collapse them into one attempt nor present them as four independent actions.
5. Check the consequences for detections and cases
A deduplication change can affect what downstream queries count. We recommend identifying threshold detections, dashboards and cases using the affected stream. For each, state whether the expected number represents deliveries, distinct source events or corroborated actions. These units are not interchangeable.
Run the synthetic scenarios through relevant detections. A genuinely repeated sequence should still produce the expected result. A retransmission alone should not manufacture that sequence when the rule is intended to count distinct events. Preserve the processing and rule versions used for comparison. Do not rewrite old case conclusions without reviewing their scope.
For an open case, explain the correction: which records were grouped, under which identity, which occurrences remain distinct and what uncertainty remains. If historical visibility cannot reconstruct deliveries, say so. An alert disappearing after a counting change does not establish that the underlying activity was benign.
6. Define evidence of a lasting recovery
Before deployment, name the processing owner, success criteria and procedure for returning to the previous version. Afterwards, monitor missing identifiers, conflicting content, arrivals outside the matching window and known recovery operations. Useful thresholds depend on the stream; this method supplies no universal deduplication duration for every source.
The intended result is an explainable decision: what is counted, what has been grouped, why and with what limitations. Review a small sample with an analyst who did not design the change. They should be able to trace a retained occurrence back to its source and explain why another delivery was grouped with it. Restored collection and reliable counting each need their own verification.