Suspicious systemd service: connect the change to actual execution
Investigate a new or modified systemd service by separating files on disk, loaded configuration, observed processes and the effects of response actions.
Integration scope: Seculogik uses a source-agnostic architecture to ingest security alerts and events through its own data map. Wazuh and OpenSearch are the first integrations. Other named sources describe planned connector targets; their availability must be confirmed for each evaluation.
An alert reports that a Linux service was created or modified. The useful first question is specific: what changed, and was that change used to execute anything? A service name, location or description cannot answer that on its own.
Our approach keeps three views separate: files on disk, the configuration known to the manager, and activity actually observed. This guide uses public systemd documentation. Check option availability against the installed version.
1. Identify the correct unit and manager
Start by recording the machine, collection time, full unit name and scope of the observation. Investigating a system unit does not automatically cover a user's units. Record the exact instance when the name contains an “@” character.
Systemd searches several locations for units. Additional files called drop-ins can change the main definition; templates and their instances also require the appropriate configuration to be examined. Do not restrict the investigation to the single file that triggered the alert. Source: systemd unit configuration.
We recommend tying each piece of evidence to the same technical identity before building a timeline. A similar name on another host, inside a container or under another user remains a separate observation until verified.
2. Capture the three views before reconciling them
systemctl cat shows files on disk, which may differ from loaded configuration. systemctl show exposes manager and current-state properties. Enablement and execution are also separate. Source: systemctl.
Read-only example for a fictional system unit, to adapt to the authorized scope:
systemctl --no-pager cat example.service
systemctl --no-pager show example.service \
-p Id -p FragmentPath -p DropInPaths \
-p LoadState -p ActiveState -p SubState \
-p MainPID -p ExecStart -p User
These are documentation examples, not tests performed in your environment. Unit contents and command arguments may contain sensitive information. Keep the outputs in the authorized investigation record and redact copies intended for wider distribution.
Do not reload configuration merely to remove a discrepancy during collection. That would be an intervention, separate from observation. Our recommendation is to preserve the useful states first, then decide which actions are needed according to urgency.
3. Examine what could actually run
ExecStartPre, ExecStart, ExecStartPost and ExecStop describe different lifecycle commands. An “active” state does not guarantee that a process is still running: RemainAfterExit can preserve that state after a command exits. Source: service configuration.
Execution context matters as well. User, WorkingDirectory, RootDirectory and environment settings can change the identity or filesystem view used by the service. Source: systemd execution environment.
We recommend comparing the observed definition with the expected deployment: service owner, package or installation procedure, intended version and approved change. Check referenced executables and scripts separately. A hash identifies content; it does not independently establish that the content is benign. Likewise, a legitimate account does not automatically explain every command running under its identity.
4. Reconstruct execution without inventing history
journalctl supports filtering by unit and time range, with timestamps displayed in UTC. User units require the corresponding filter and context. Source: journalctl.
For a fictional system unit, the collection pattern is:
journalctl --no-pager --utc -u example.service \
--since "2026-10-31 08:00:00 UTC" \
--until "2026-10-31 09:00:00 UTC" -o short-iso-precise
Choose boundaries appropriate to the investigation; this window is illustrative only. Preserve original timestamps and useful identifiers in evidence exports.
The journal defines fields for the system boot and unit invocation, among others. These help distinguish executions that a shared unit name or reused PID might otherwise confuse. Source: journal fields.
Compare these records with the process, authentication, deployment and network evidence actually available. Document each connection. A file modification timestamp does not identify its author; current state does not reconstruct all past activity.
Journal retention depends on configuration, including volatile or persistent storage and retention limits. Source: journald.conf. An empty search therefore needs a statement of scope and limitations rather than being turned into proof that nothing ran.
5. Decide on a response and verify its effects
We recommend basing the decision on established behavior and business impact. A change explained by a verified deployment, confirmed suspicious execution and activity that cannot be reconstructed call for different treatment.
Before intervening, define the intended outcome: prevent a new execution, stop current activity, restore configuration or preserve evidence. Examine mechanisms that could restart the service and its business dependencies. A successful administrative command does not prove that every intended outcome has been achieved.
After the authorized action, revisit all three views: files, loaded configuration and observed activity. Confirm that collection still works. Where history remains incomplete, retain that limitation in the verdict.
A useful conclusion fits in a few lines
The final record should answer five questions: what change is established, what execution is proven, which explanation remains a hypothesis, what action was taken, and what check confirms its effect? Add the next decision, its owner and the missing evidence.
This method applies regardless of which tool raised the alert. Its value comes from the connection between the initial signal, observations and decision, rather than the number of screenshots collected.