Your system logs everything: actor, input, tool invoked, action, approval, result, time. Seven impeccable fields, exportable, with timestamps. The IT area shows them with pride when someone asks about traceability.
And on the day of the incident you discover that those seven fields answer a single question, what happened, and that the question of the incident is another one: why. The day you need the difference will be the worst day to discover that you did not have it.
Auditing AI systems is not accumulating logs. It is being able to reconstruct the causes of a decision, with the depth the risk of the position demands. That discipline has a name in the book, chapter 7: auditing is not logging.
The why lives in the determinants
The determinants of a decision are the conditions that produced it: what policy was in force when the action was executed, what knowledge the system consulted, what its tools returned to it, what authority it had at that moment and what versions of all of the above were operating.
Without the determinants, a failure cannot be attributed to its cause. And the possible causes are at least five, each one with a different remedy: a badly written policy, knowledge that aged without anyone updating it, a tool that returned bad data, a model that erred, or a role that should never have been assigned to an artificial resource. Correcting the first one is rewriting a document. Correcting the last one is returning the position to human hands. They do not resemble each other at all.
The problem is that under an only-what log, those five causes leave identical traces. The postmortem chooses one at random, the organization "corrects" the wrong one, and the real failure stays intact waiting for its second chance. With the same policy, the same old data or the same broken tool still in place.
Four stories, a single log entry
The book grounds it with the collections thread that runs through its chapters. The role sent a payment reminder to an account in dispute: exactly what its role contract forbids. Why?
Maybe the dispute flag did not exist in the system when it consulted: a data problem. Maybe it existed, but the policy that defines "dispute" changed last week and its context had the old version: a knowledge problem. Maybe the query tool returned the outdated status: an integration problem. Maybe everything was fine and the model ignored it: a model problem. Four stories, four culprits, four different remedies.
And a single log entry: "message sent at 9:14". The audit of determinants is what separates those four stories. The log, however impeccable it may be, cannot.
The deep reason for reconstructing determinants is not technical but administrative: every cause points to a different person responsible. The badly written policy belongs to whoever drafted it. The old knowledge belongs to whoever had to update it. The broken tool belongs to whoever integrates it. Without determinants, the blame falls on whoever is closest to the incident. The book closes the chapter with the exact rule: blame follows the determinants, not the proximity.
The model's account is not the audit
There is a modern temptation that deserves a specific warning: asking the model why it did it. Current models narrate their reasoning with a charming eloquence. They apologize, they point out the step where they "got confused", they propose how to avoid it. It is convincing. And it is inadmissible as proof.
The reason is uncomfortable but well established: what a system reports having thought may not be what produced its output. The narration is one more output of the same model, generated afterwards, not a window into its mechanism. The book admits it as support, never as proof. An organization that takes the model's account as the why is going to write confident and mistaken postmortems, which are the most expensive of all: they close the investigation with the incorrect cause and the certainty of having found it.
The serious auditor's answer does not change with technology: do not ask the suspect, reconstruct the scene. Policy in force, knowledge consulted, response of the tools, authority granted, versions operating. Nobody narrates that. It is evidenced or it does not exist.
The version manifest
The piece that makes all of this operable is humble: the version manifest. Model, policies, tools and knowledge base, each one with an identifiable version at all times. Without it, it is impossible to say what authority existed at a given moment, nor what change produced an improvement or a deterioration. The standard requires it in clause HWF-43, which fits in one line: "An AI Employee must have an identifiable version of model, policies, tools and knowledge base."
And here comes a relief for the small operator, because the burden is proportional to the risk. In positions of Low and Moderate risk, correlating the event logs against the version manifest is sufficient and conforming reconstruction. Only from High upwards are the determinants tied one by one to each action. The standard does not ask you for laboratory instrumentation for a role that sends reminders; it asks you to know what version of what thing was alive when something important happened.
The complete obligation lives in HWF-41: every material action must be auditable, and the audit must be able to reconstruct the policy, the knowledge, the results of the tools, the authority and the versions in force at the moment the action was executed. It is not bureaucracy. It is the difference between correcting the cause and correcting at random.
What works can also stop working
The audit does not exist only for the day of the incident. It exists because performance is not an eternal property of the system: it is a relationship between the resource, the position and the context. And the context does not sign stability contracts. Products, clients, policies, data, costs and vendors change. A technical update can improve one capability and affect another.
That is why the evaluation does not end when the pilot goes well. It is worth watching trends: a human intervention rate that rises slowly can reveal degradation before the visible incident, because drift, the gradual deviation of behavior with respect to its validated performance, rarely announces its arrival with a spectacular error. A cost per result that rises can make senseless a configuration that keeps its precision.
And when something material changes (the model, the tools, the policies, the knowledge, the authority), the role is revalidated before operating, at the depth its risk class demands. Documenting the change does not demonstrate that it works: demonstrating it is a separate act, and the order matters. First the evidence, then the operation.
A version is a record, not a proof.
A test you can do today
Choose a real action that your most important automation executed last week. A concrete one. Can you say what version of what policy was in force at that moment? What data it consulted and what was returned to it? What authority it had?
If the answer is "I would have to ask the vendor", your audit lives in the hands of someone who does not answer for your consequences. The vendor does not sign your postmortems, does not face your damaged client and does not explain to the board why the failure repeated itself. You do. The capacity to reconstruct the why has to live on your side of the counter, sized to the risk of each position.
Logging is cheap and that is why everybody does it. Auditing is reconstructing. And it is designed beforehand.
Frequently asked questions
Auditing is being able to reconstruct the determinants of a decision, not only its result: what policy was in force when the action was executed, what knowledge the system consulted, what the tools returned, what authority it had and what versions of all of the above were operating. A traditional log (actor, action, result, time) only answers what happened; the audit answers why, which is the question of the day of the incident. Without that capacity, different causes leave identical traces, the postmortem chooses one at random and the organization corrects the wrong one while the real failure waits for its second chance.
They are the conditions that produced the decision: the policy in force at the moment of executing the action, the knowledge the system consulted, the results its tools returned to it, the authority it had been granted and the versions of model, policies, tools and knowledge base that were operating. They matter because the possible causes of a failure are at least five (a badly written policy, aged knowledge, a tool with bad data, an error of the model, or a role that should never have been assigned to an artificial resource) and each one has a different remedy and a different person responsible. The determinants are what allows attributing the failure to its real cause.
As support, yes; as proof, never. Current models narrate their reasoning with eloquence, but what a system reports having thought may not be what produced its output: the explanation is one more output of the same model, generated afterwards, not a window into its mechanism. An organization that takes that narration as the why writes confident and mistaken postmortems, the most expensive of all, because they close the investigation with the incorrect cause. The alternative is to reconstruct the scene with evidence: policy in force, knowledge consulted, responses of the tools, authority and versions at the moment of the action.
It is the record that keeps identifiable, at all times, the version of the model, of the policies, of the tools and of the knowledge base with which an AI Employee operates. Without it, it is impossible to say what authority existed at a given moment nor what change produced an improvement or a deterioration. It makes the audit operable with a burden proportional to the risk: in positions of Low and Moderate risk it is enough to correlate the event logs against the manifest; from High upwards, the determinants are tied one by one to each action. The standard requires it in one line: model, policies, tools and knowledge base with an identifiable version.
The missed escalations that the audit uncovers feed the AI Employee scorecard. And when the determinants show that the position is no longer justified in artificial hands, the following rule applies: no role lasts forever.
Want the full method? Read AI Employee. For executive AI consulting or keynotes and workshops.
Go deeper
Want to bring your team to the next belt?
Book a discovery call or explore the full book.