SOC API exports: verify pagination before drawing conclusions
A successful request does not establish a complete export. Check pagination, accessible scope and recovery before concluding that an event is absent.
Integration scope: Seculogik uses a source-agnostic architecture to ingest security alerts and events through its own data map. Wazuh and OpenSearch are the first integrations. Other named sources describe planned connector targets; their availability must be confirmed for each evaluation.
An analyst requests events for a particular period. The API responds successfully, a file is created and a search of that file finds nothing suspicious. But does the file contain every accessible result, or just the first page? A successful request and a completed extraction are different findings.
This guide proposes a provider-independent verification method for collection pipelines, investigation exports and assistance tools that query APIs. The documented examples make no claim about available Seculogik connectors. Our recommendations concern the quality of retrieved evidence, without requiring broader permissions or unnecessary data exports.
1. Define what the export must cover
We recommend writing down the search contract before extraction. Specify the organisation, target resources, event family, period and timezone, filter and technical identity used. Identify the time field being queried: event time or receipt time, for example. Thoroughly traversing a period against the wrong field may still fail to answer the investigation question.
Then separate three conclusions: the request was accepted; traversal finished under the API’s contract; and the results cover the evidence the investigation needs. The second does not establish the third. Effective permissions, filters, retention and unavailable sources may limit what can be retrieved.
If the need exceeds authorised access, document the limitation and refer the request to the appropriate owner. Do not broaden permissions merely to make an empty result disappear. The case should explain what was searched and what could not be searched.
2. Identify the documented continuation signal
For applicable paginated responses, Microsoft Graph supplies an @odata.nextLink property to follow until it is no longer returned. Its documentation calls for using the entire returned URL rather than extracting a fragment to build a different request. Exact behaviour depends on the API being queried. Source: Microsoft Graph
GitHub REST illustrates a different contract: pagination links appear in the link response header, with the next relation identifying the following page. Processing that examines only the response body can therefore miss continuation information. Source: GitHub Docs
For each operation used, we recommend recording where the signal appears, how to follow it and which state actually means “finished”. Check the behaviour of your library or connector. A function returning a list does not establish that it automatically traversed every page.
Google’s AIP-158 guidance even permits an empty page before the end, with continuation available. A short or empty page is therefore not a universal end-of-collection signal. This document is API design guidance; check the specific service contract before assuming that behaviour applies. Source: Google AIP-158
3. Preserve a record of traversal and interruptions
We recommend a proportionate execution record: search reference, start and finish, pages received, records saved, errors and reason for stopping. Distinguish normal completion from a timeout, local cap, access refusal, expired cursor or operator intervention. A partial file can still be useful when clearly identified as partial.
A checkpoint should correspond to records actually saved. If processing advances its cursor and then fails before persisting the data, recovery can skip those records. If it restarts a page already saved, that page may be delivered twice. Test both situations instead of assuming a restart repairs everything.
Continuation links and cursors may contain sensitive information. Keep values needed for recovery in authorised storage; do not copy them into public comments or every diagnostic log. For routine tracking, a controlled internal reference and a “continuation present” state may be sufficient.
Also define an execution limit to protect against continuation loops, followed by an explicit failure state if the limit is reached. Such a cap protects execution. It must not silently convert an interrupted extraction into a complete export.
4. Account for data changing during traversal
Elasticsearch documentation explains that a refresh between successive searches can change result ordering. A point in time can preserve index state during traversal; sorting and tie-breaking requirements still apply. This facility belongs to Elasticsearch and supplies no guarantee for another API. Source: Elastic search pagination
Our recommendation is to determine whether the source offers a stable view, consistent continuation or a list that can change. Fixing a search end boundary helps define the request but does not necessarily freeze the data. Late arrivals, modifications or deletions may still affect the result.
Without a stable-view guarantee, document the limitation and agree an appropriate reconciliation method with the source owner. A second pass can help find discrepancies; it does not automatically establish completeness. If you overlap periods to account for late arrivals, verify event identity before deduplicating.
5. Test paths that can produce false success
Before accepting a collection pipeline or export, we recommend controlled tests using synthetic data containing no secrets. Write down expected outcomes, then check at least these situations:
- Several pages are required, with a known test event on the final page.
- A short or empty page still carries continuation when the contract permits it.
- An error occurs after a page has already been saved.
- Persistence fails before the checkpoint is committed.
- A cursor expires or a continuation link repeats.
- Data changes during traversal under the source’s documented behaviour.
Entirely fictional example: a test dataset has three pages, with a known test event only on the third. An initial export times out after the second page. It must be marked partial and cannot support a finding that the test event is absent. After controlled recovery, check that the event is present, the expected pages have been retained and a replayed page has not created an artificial extra occurrence.
Do not assess the test solely by its final volume. Verify the test event’s identity, time boundaries, covered resources and stopping reason. Also check that data from another organisation has not been included. A plausible total can conceal a gap, duplicate or mixed scope.
6. Write a conclusion that respects the scope
In the case, record extraction status separately from the analytical finding. A defensible statement might be: “No matching event in the pages retained for this search; extraction stopped before the end signal.” Even after completed traversal, state the period, filters and known access or visibility limitations.
Export acceptance criteria should allow another analyst to retrieve the request, understand the stopping condition and verify a sample of results. A file existing or a command finishing successfully is not sufficient. Name the owner of unresolved extraction work and preserve the condition for reassessing the finding if further pages become available.
For related controls, see our guide to duplicate logs and retransmissions. Traversing every accessible page and counting events correctly are complementary checks.