AI agent forensics: how to prove what your AI actually did
Here's a question most AI app security tools can't answer: last Tuesday, your agent deleted a customer's records. Was it a bug, a bad instruction, a prompt injection, or working exactly as designed? A vulnerability scanner shrugs — it looks for holes before something happens, and has nothing to say about what your agent already did. Proving what an autonomous system did, after the fact, is a different job than blocking it beforehand — and most software security has barely built for it. For AI agents, that gap is about to be a serious problem.
Why agents break the old model
A traditional web app is predictable: it does what its code says, so you can read the code, scan it, and reason about its behaviour. An AI agent is different in three ways that matter for forensics.
It's non-deterministic — give it the same situation twice and it may act differently, so you can't reproduce a bug by re-running it, and the code didn't decide, the model did. It takes real actions — modern agents call tools, hit APIs, move data, and spend money, increasingly over MCP (the Model Context Protocol) to servers you don't fully control. And its instructions are attackable — prompt injection and tool-spec injection can change what your agent does without touching a line of your code. Put those together and you get a system that can do something harmful, leave no trace in the code, and never do it the same way twice.
What “AI agent forensics” actually means
Agent forensics is capturing what an AI agent did in a form complete and trustworthy enough to reconstruct any incident after the fact. It has four requirements.
Completeness — every consequential action, input, output, and the context the agent worked from is recorded; gaps are where accountability dies. Tamper-evidence — if logs can be quietly edited, they prove nothing; a tamper-evident record makes alteration detectable. Reconstruction — raw logs aren't forensics; forensics is replaying the sequence into a clear narrative of what happened and why. And feedback into prevention — the point of reconstructing an incident is to harden the system so it can't recur.
“I'll just add logging” — why that's not enough
Most teams assume they already have this. They have console.log. That isn't forensics, for the same reason a shoebox of receipts isn't accounting. Ordinary application logs are written by the same app you're trying to investigate, in whatever shape the developer chose, with no guarantee of completeness and no protection against being changed. When you actually need them — during an incident, a dispute, or an audit — you find the one thing you needed wasn't logged, or you can't prove the logs weren't altered. Forensics is logging built for the moment someone asks “prove it,” not for debugging on a normal day.
Why this is becoming non-optional
Two forces are turning agent forensics from nice-to-have into a requirement. Regulation: the EU AI Act's Article 12 requires certain AI systems to keep automatic records of their operation over their lifetime, specifically so events can be traced and reconstructed. If your agent is in scope, “we didn't keep records” stops being an oversight and becomes a compliance failure.
Trust: as agents take more autonomous action with users' data and money, “trust me, it's fine” stops working. Your customers, investors, and your own team will want proof of what an automated system did. The teams that can produce that proof will win the ones that can't.
Where Opviva fits
Most security tools — including most of the vibe-coding scanners that appeared this year — hand you a list of maybe-issues and stop there. Opviva is built differently: it's a security agent you talk to. You describe what you shipped, it scans your live app and code, and when it flags something, it proves the vulnerability is real by reproducing the exploit — the same proof-by-reconstruction this article is about, turned on your own agents. Every step is recorded on a tamper-evident Evidence Canvas, so “we think this might be exploitable” becomes “here's exactly what happened, and how” — and then it opens the fix as a pull request you approve, and keeps watching after launch. Talk, it proves, it fixes, it watches.
If you're shipping AI agents, ask yourself one question: if your agent did something it shouldn't have, could you prove exactly what happened? If the answer is “I'd be guessing from incomplete logs,” you have a proof gap — and a tool that stops at a report won't close it.
Frequently asked questions
What is AI agent forensics?
AI agent forensics is capturing what an AI agent did — its prompts, tool calls, decisions, and the data it touched — in a tamper-evident, complete form you can reconstruct after the fact, so you can prove what happened during an incident, dispute, or audit.
Isn't application logging enough?
No. Ordinary logs are written by the same app you're investigating, in an arbitrary shape, with no guarantee of completeness and no protection against being altered. Forensics is logging built for the moment someone asks “prove it” — tamper-evident and reconstructable — not for everyday debugging.
Does the EU AI Act require this?
The EU AI Act's Article 12 requires certain (high-risk) AI systems to automatically keep records of their operation over their lifetime so events can be traced and reconstructed. Whether your system is in scope is a determination for you and your counsel — but if it is, reconstructable record-keeping is a legal requirement, not optional.