Agent crash, OOM kill, bad deploy, unauthorized access — anything unexpected.
SHA-256 audit ledger captures timeline, root cause, and recovery actions.
"What happened at 3am?" — OpenClaw surfaces the full incident.
Tamper-Proof Audit Ledger for Incident Forensics
Every action your AI agent takes is recorded in a SHA-256 hash-chained audit ledger. Tool calls, process restarts, configuration changes, mesh communications, secrets access -- all cryptographically sealed and permanently recorded. When something goes wrong, the ledger tells you exactly what happened, when it happened, and why. No guesswork. No missing logs. No tampered evidence.
Incident forensics for AI agents is fundamentally harder than for traditional software. Traditional applications follow predictable code paths -- when they fail, you read the stack trace and find the bug. AI agents make decisions based on LLM reasoning that varies with context, temperature, and prompt content. The same agent can behave differently on consecutive runs with identical inputs. When an agent produces an unexpected output, causes data corruption, or takes an unauthorized action, the investigation requires a complete, chronological record of everything the agent did -- not just the final error, but the entire chain of decisions and actions that led to it.
In 2026, with AI agents handling real business operations -- customer interactions, financial transactions, data processing, infrastructure management -- the stakes of incident investigation are higher than ever. A support agent that gives incorrect information, a data agent that corrupts records, a deployment agent that misconfigures a service -- these incidents require not just technical debugging but often regulatory reporting, customer communication, and sometimes legal proceedings. In all of these contexts, the integrity of the forensic record is paramount. The osModa audit ledger provides that integrity through cryptographic hash chaining at the platform level.
TL;DR
- • Every agent action is recorded in a SHA-256 hash-chained ledger at the platform level, not the application level
- • Captures tool calls, crashes, watchdog recoveries, config changes, SSH sessions, secrets access, and mesh events
- • Hash chain verification proves forensic evidence has not been tampered with -- critical for legal and compliance contexts
- • Structured entries are filterable by time, agent, action type, and severity for rapid incident investigation
- • The same ledger powers both incident forensics and compliance evidence for SOC2, HIPAA, and 21 CFR Part 11
Incident Investigation Workflow
When an incident occurs, the audit ledger provides a structured workflow for investigation. Here is how you go from “something went wrong” to “here is exactly what happened and why.”
- 1
Identify the Time Window
Start with the known time of the incident (the customer complaint timestamp, the alert trigger time, or the first observed symptom). Query the audit ledger for events in a window around that time. The structured entries let you quickly scan for anomalies: unexpected tool calls, error responses, crashes, or recovery events.
- 2
Trace the Action Chain
Follow the agent's actions backward from the incident. Each audit entry links to the previous entry through the hash chain, creating a verifiable chronological sequence. You can see every tool call, every API response, every decision point that led to the incident. For multi-agent incidents, trace across agents using mesh communication events that appear in both agents' ledgers.
- 3
Verify Evidence Integrity
Run the hash chain verification to confirm that no entries have been modified between the incident and your investigation. This is particularly important when the investigation has compliance or legal implications. The verification proves that the forensic record is authentic and unaltered, which standard logs cannot provide.
- 4
Export and Report
Export the relevant ledger entries as structured JSON for incident reports, compliance filings, or post-mortem documentation. The export includes hash chain verification metadata so stakeholders can independently verify the evidence. The structured format makes it straightforward to include in formal incident reports.
What the Audit Ledger Captures
The audit ledger captures events at the platform level, providing forensic depth that application-level logging cannot achieve.
Tool Call History
Every invocation of the 66 built-in tools is logged with input parameters, output results, execution duration, and calling agent identity. For incident forensics, this is the most valuable data: you can see exactly what the agent did, in what order, with what parameters, and what results it received. File reads, HTTP requests, process spawning, secrets access -- every external action is documented.
Recovery Events
The watchdog daemon logs every crash detection, restart attempt, health check result, and escalation action. For incident forensics, this tells you whether the agent crashed, how many times it crashed, how quickly it recovered, and whether the recovery was successful. Patterns in recovery events (repeated crashes, escalations to rollback) often reveal the root cause of incidents.
Configuration Changes
Every NixOS generation switch is logged with old and new configuration hashes and the change diff. This is critical for correlating incidents with recent deployments. If an agent starts misbehaving after a deployment, the audit ledger shows exactly what changed and when. Combined with NixOS rollback capability, you can both identify and remediate configuration-related incidents quickly.
Access and Authentication
SSH sessions, secrets access, and mesh network authentication events are logged with actor identity and timestamp. For security incidents, this tells you who had access to the system, when they accessed it, and what they did. The tamper-evident hash chain ensures that access records cannot be modified after the fact, even by the people being investigated.
Real-World Forensic Scenarios
Here are concrete scenarios where the audit ledger's forensic capabilities provide real value.
Agent Produces Incorrect Output
A customer-facing AI agent provides incorrect information that leads to a complaint. You query the audit ledger for the agent's tool calls during the interaction window. The ledger shows: the agent called an external API that returned stale data, the agent used that data in its response, and no error was detected by the agent's logic. Root cause: external API returned cached data. The tamper-evident record proves the investigation examined authentic evidence, which matters for customer communication and potentially regulatory reporting.
Repeated Agent Crashes
Your monitoring shows that an agent has crashed and been restarted by the watchdog 12 times in the past hour. You query the audit ledger for recovery events and the tool calls immediately preceding each crash. The pattern emerges: the agent crashes every time it attempts to call a specific tool with a particular parameter range. The ledger provides the exact crash sequence, enabling you to reproduce and fix the bug. The recovery timeline shows that the watchdog maintained near-continuous availability despite the crashes.
Unauthorized Data Access
A security audit reveals that sensitive data was accessed from your agent server. You query the audit ledger for all access events during the relevant time window. The ledger shows: an SSH session from an unexpected IP address, followed by secrets retrieval and data file access. The hash chain proves these records have not been modified since they were created. This forensic evidence supports your incident response plan and breach notification obligations.
From Forensics to Compliance
The same audit ledger that powers incident forensics also generates compliance evidence. The structured, tamper-evident records satisfy audit requirements across multiple frameworks.
Frequently Asked Questions
How is the audit ledger different from standard application logs?
Standard application logs (stdout, syslog, journald) are mutable, free-text, and application-controlled. An admin can delete entries, an application can choose not to log, and there is no way to verify that logs have not been tampered with. The osModa audit ledger operates at the platform level (below your application code), captures structured typed events, uses SHA-256 hash chaining for tamper evidence, and cannot be silently modified by application code or administrators.
What events are captured in the audit ledger?
The ledger captures: all 66 tool call invocations and their results, agent process starts/stops/crashes, watchdog recovery events, NixOS generation switches, SSH session events, secrets access, mesh network connections, health check results, resource utilization snapshots, and configuration changes. Every event includes a UTC nanosecond timestamp, actor identity, action type, structured payload, and SHA-256 hash linking to the previous entry.
How does the audit ledger help with incident investigation?
After an incident, you can query the ledger for the time window around the event. The structured entries let you reconstruct the exact sequence of actions: what the agent did, what tools it called, what responses it received, when it crashed, how the watchdog recovered it, and what state the system was in throughout. Because the entries are tamper-evident, you can trust that the forensic record has not been altered.
Can I search and filter the audit ledger?
Yes. The ledger supports filtering by time range, actor identity (specific agent, user, or daemon), action type (tool calls, restarts, config changes), event severity, and custom metadata fields. You can also perform full-text search across structured payload data. Filters can be combined to narrow down to the exact events relevant to your investigation.
How does the hash chain work for incident forensics?
Each entry's SHA-256 hash incorporates the previous entry's hash, creating a chain. During an investigation, you can verify the chain integrity to confirm that no entries have been added, removed, or modified between the incident and your investigation. This is crucial when the investigation has legal or compliance implications -- you need to demonstrate that the forensic evidence is authentic and unaltered.
Can I export ledger data for external analysis?
Yes. The ledger supports export in structured JSON with hash chain metadata. You can feed the exported data into external analysis tools, SIEM systems, or data warehouses. The export includes verification instructions so external tools can validate chain integrity. Exports can be filtered to specific time ranges or event types to keep the data volume manageable.
What happens to the audit ledger during a NixOS rollback?
The audit ledger is stored in a data directory that is independent of NixOS generations. A rollback reverts system configuration but does not affect the audit ledger. This is by design: you need the audit trail to survive rollbacks so you can investigate what went wrong with the rolled-back configuration. The rollback itself is recorded as an event in the ledger.
Is the audit ledger included on all plans?
Yes. The tamper-proof audit ledger is included on every osModa plan from $14.99/month to $125.99/month. There are no premium tiers or add-ons for incident forensics capabilities. Every plan includes all 9 Rust daemons, including the audit writer daemon.
Know What Your Agents Did. Prove It.
Every osModa plan includes the tamper-proof audit ledger for incident forensics and compliance evidence. From $14.99/month.
Last updated: March 2026