Document withdrawn from a RAG assistant: verify access and remaining copies

Integration scope: Seculogik uses a source-agnostic architecture to ingest security alerts and events through its own data map. Wazuh and OpenSearch are the first integrations. Other named sources describe planned connector targets; their availability must be confirmed for each evaluation.

A sensitive procedure has been withdrawn from an assistant's document collection. Its link no longer works, and a new question does not cite it. Does that establish that it is no longer accessible? Not without examining indexed chunks, permissions applied during retrieval and previously retained responses.

RAG enriches a model's responses with external sources retrieved for the request. OWASP identifies inadequate access controls and cross-context leakage among the risks affecting vector retrieval used for this augmentation. Source: OWASP LLM08:2025

This guide proposes an operational withdrawal check independent of model and search engine. Documented examples include Azure AI Search; they do not imply an available Seculogik integration. The proposed tests use a synthetic corpus without incident content or production data.

1. Define exactly what “withdrawn” means

Our recommendation is to start with the required decision. Must a person lose document access, must the assistant stop using it, must an obsolete version be replaced, or must data be deleted within a defined scope? Those objectives do not necessarily require the same operations.

Identify the source, version, owner, affected profiles and decision time. List the relevant checkpoints: document store, retrieval, model context, displayed citations and retained responses. Assign an owner and an observable outcome to each applicable checkpoint.

Also separate removal from retrieval from authorised retention for an investigation. A protected archive can serve a distinct need; it should not become a searchable source again by default. Retention rules and any erasure obligations belong with authorised owners.

Avoid a conclusion such as “the model forgot”. This verification concerns content access and reuse in the system examined. If data also served training or another processing purpose, open a separate assessment of that scope.

2. Connect the source to its chunks and versions

Microsoft explains that, in some Azure AI Search ingestion modes, deleting a source before the expected detection step can leave orphaned documents in the index. Its documentation distinguishes source deletion from deletion of search documents, including chunks. This behaviour depends on the indexing mode in use. Source: Microsoft, deleting documents from an index

We recommend requesting an inventory of source-derived material: chunks, previous versions and any secondary indexes. Verify their relationship with the original document using stable identifiers. A title search is insufficient if the title changed or the chunks do not contain it.

Preserve the observed state and authorised operation results, with timestamps. Then check the outcome through the interfaces actually used by the assistant. A successful deletion request does not itself demonstrate disappearance from every path.

Finally, check the next ingestion cycle. Could an old copy or a still-present source reintroduce the document? Withdrawal criteria should address recurrence, rather than one moment when a query returns nothing.

3. Check authorisation before content reaches the model

OWASP recommends validating permissions on every request rather than relying on checks covering most paths. For this verification, that means testing retrieval and direct object access under the relevant profile, rather than checking only the visible interface. Source: OWASP, authorisation controls

Microsoft's security-filter example is instructive: a string representing a user or group filters results; it does not authenticate that identity. The documentation also states that hiding a field in a response is not a content-security mechanism. Source: Microsoft, security filters in Azure AI Search

Our recommendation is to establish which identity actually participates in the access decision and where its permissions come from. A filter displaying the correct name is insufficient. Have someone authorised to inspect service behaviour confirm the control, without copying secrets into the case.

Denial should occur before a forbidden chunk enters model context. An instruction asking the model not to reveal it does not replace authorisation. Check titles, excerpts and citations too: removing the main text should not leave another response exposing the protected information.

4. Examine caches and existing conversations

OWASP's RAG guidance recommends isolating caches according to permissions and invalidating them when sources or permissions change. It also calls for propagating source deletion to relevant derived data. Source: OWASP, RAG security

We recommend comparing a question never previously asked with one that already produced an answer. Test a conversation opened before withdrawal and a new conversation as well. Record whether a response came from fresh retrieval, a cache or history where the system makes this observable.

Treat previously visible responses as separate objects for review. Source withdrawal does not, by reasoning alone, erase an exported copy or an earlier message. Define what should remain accessible, to whom and with what version warning, then have that decision applied within the controlled scope.

Where a response's production path is unobservable, record that limitation. Text resembling the document does not by itself establish whether it came from recent retrieval, history or independent information.

5. Design a test that can actually fail

Our recommendation is to prepare a fictional document containing a distinctive detail with no sensitive value, together with a control document that should remain available. Define expected outcomes before testing. A test in which no profile retrieves anything could simply indicate a search outage.

With authorised test identities, cover these situations:

1. Before withdrawal, the intended profile retrieves the synthetic source and its distinctive detail.
2. After withdrawal or revocation, that profile no longer receives forbidden chunks, even when asking for the detail differently.
3. The control document remains available to an authorised profile: the check does not rely on a general outage.
4. A previously used question and an older conversation do not re-expose content outside the agreed scope.
5. The next ingestion cycle does not reintroduce the withdrawn version.

Stay within the system and corpus authorised for the exercise. Do not try to obtain another team's actual documents to test isolation. A synthetic marker lets you observe an outcome without creating a new disclosure.

Several phrasings can help explore variable behaviour, but they do not prove a universal absence of leakage. Combine tests with available retrieval observations, authorisation decisions and lifecycle controls.

6. Close with withdrawal evidence, not a reassuring answer

Entirely fictional example: a new conversation no longer cites the withdrawn procedure, but a previously used question returns an old retained response. This establishes incomplete withdrawal along a reuse path; it does not establish that every user received the content.

We recommend preserving the scope, test profiles, identified source and derivatives, operations performed and before-and-after outcomes. Prefer necessary references and traces to copying sensitive content in full. Make unobservable paths and remaining exceptions visible.

If forbidden material is found, obtain authorisation for a proportionate measure: restricting the affected path, correcting propagation or addressing relevant copies. Preserve useful evidence before changes, without maintaining an urgent exposure merely to perfect collection.

Withdrawal is verified when agreed checks pass within the stated scope. Our free investigation resources can help structure this case, clearly distinguishing the decision, a successful operation and an actually observed effect.