Governance that engineers can live with

The questionnaire has 84 questions and a two-week deadline. It concerns a credit decisioning model that has been live for thirteen months.

Question 11 asks which data the model was trained on. The answer is a table that has since been backfilled twice. Question 26 asks who approved the deployment. There is a Slack thread with a thumbs up in it. Question 40 asks for the error rate by customer segment. Nobody has ever computed one. Question 63 asks what happens when the model is unavailable, and two engineers give different answers in the same meeting.

The team spends three weeks writing a document. Much of it is written from memory, which means parts of it are wrong, and everybody involved knows which parts and says nothing. The document is filed. Nothing about the system improves.

This is what most people mean when they say governance, and it is why engineers dislike it. It arrives late, it asks for evidence that was never captured, and it produces prose rather than change.

There is a version that works, and the difference is not that it asks less. It is that every answer is a byproduct of work the team was already doing.

What governance actually has to produce

Strip away the vocabulary and four questions remain. Every framework worth the name is a way of being able to answer them.

What does this system do, and to whom? Which decision, at what volume, affecting which people.
How well does it work, and where does it work worst? Not one number. A number per segment that matters.
Who decided it should run, on what evidence, and when?
What happens when it is wrong, and who acts?

The team above could answer none of them in two weeks. A team that captured them as it went could answer all four in an afternoon, because each answer already exists somewhere. The scope page from The first week of an AI project answers the first. The evaluation attached to a run record answers the second. The deployment record from Deploying a model is not the finish line answers the third. The alerting and on-call arrangement in Monitoring for silent failure answers the fourth.

Governance that engineers can live with is mostly an indexing problem.

Documentation that is generated, not written

Two pieces of prior work turned documentation from an essay into a schema, and both are worth adopting directly rather than paraphrasing into a house template.

Mitchell, Wu, Zaldivar, Barnes, Vasserman, Hutchinson, Spitzer, Raji and Gebru proposed model cards: short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups and intersectional groups that are relevant to the intended application domains. Their reasoning was to clarify the intended use cases of machine learning models and minimize their usage in contexts for which they are not well suited, and their cards also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information.

Gebru, Morgenstern, Vecchione, Wortman Vaughan, Wallach, Daume and Crawford did the same for data, observing that the machine learning community currently has no standardized process for documenting datasets, which can lead to severe consequences in high-stakes domains. Their analogy is the useful part: in the electronics industry, every component, no matter how simple or complex, is accompanied with a datasheet that describes its operating characteristics, test results, recommended uses, and other information, so every dataset should be accompanied with a datasheet that documents its motivation, composition, collection process, recommended uses, and so on.

The engineering move that makes both survive contact with a deadline is to generate them. A model card whose evaluation table is written by hand will be stale by the third release. A model card whose evaluation table is rendered from the run record described in Experiment tracking for people who forget is correct by construction, and the only hand-written parts are the ones that genuinely require a human: intended use, known limitations, and the groups where performance is worst.

Write those three sections carefully and honestly, because they are the sections a reader will use to decide whether to trust the system, and the discipline of stating them is the same discipline Reading a model you did not build asks a reviewer to apply from the outside.

The review that is worth attending

Most review boards fail in one of two directions. They review everything, become a queue, and approve 39 of 40 items, at which point they are a rubber stamp with a calendar invite. Or they review nothing until something goes wrong, and then review that one thing forever.

Measure your own. If the approval rate is above 95 percent, the review is not doing work and should either be narrowed or given teeth. That is a governance metric, and it is the only one that reliably predicts whether the process is real.

Three moments deserve a review, and nothing else does.

Scope, in week one. The decision, the baseline, the population affected, and the refusals. Cheap to hold, and the only point at which the answer can still be "we are not building this".
Before the ramp. The evaluation by segment, the rollback trigger, the on-call owner, and the failure behaviour. This is where the production readiness question belongs.
Quarterly, while it runs. What drifted, what was retrained, what the override rate is, and what the system cost against what it returned.

Each produces one artefact and one recorded decision with a name attached. Raji, Smart, White, Mitchell, Gebru, Hutchinson, Smith-Loud, Theron and Barnes make the same structural argument for internal auditing, and their framing of why it has to be internal is the sharpest version of the case: it remains challenging for practitioners to identify the harmful repercussions of their own systems prior to deployment, and, once deployed, emergent issues can become difficult or impossible to trace back to their source. Their framework supports development end-to-end, applied throughout the internal organization development lifecycle, and each stage of the audit yields a set of documents that together form an overall audit report.

Documents at each stage rather than an essay at the end. That is the whole design.

The audit trail is a log

The question "why did the system do that to this customer on 4 February" is answerable or it is not, and the answer is decided a year in advance by what you chose to write down.

For every decision, six fields:

The input, or a reference to an immutable snapshot of it.
The model version, meaning the artefact, the transformer and the contract as one identifier.
The score.
The threshold in force at that moment.
The action taken.
The actor, whether a person or the system, and whether a human overrode it.

Price it. At around 400 bytes a record and 1.44 million decisions a month, that is 576 megabytes a month, under 7 gigabytes a year. Storage at that scale is a rounding error against the three weeks the team above spent writing a document from memory, and it converts an archaeology exercise into a query.

Two properties make the difference between a log and evidence. It has to be append-only, because a trail that can be edited proves nothing. And the threshold has to be recorded per decision rather than looked up later, because thresholds move, as The cost of a wrong answer sets out, and a record that says 0.50 today cannot tell you what was in force in February.

Where the frameworks fit

Formal frameworks are useful as a checklist of what you forgot, and harmful when adopted as a project in themselves.

The NIST risk management framework organises the work into four functions, and the phrasing rewards a careful read. Govern cultivates and implements a culture of risk management within organizations designing, developing, deploying, evaluating, or acquiring AI systems. Map establishes the context to frame risks and identify potential negative impacts across the lifecycle. Measure employs quantitative, qualitative, or mixed-method tools, techniques, and methodologies to analyze, assess, benchmark, and monitor AI risk. Manage allocates risk resources to mapped and measured risks and implements plans to respond to, recover from, and communicate about incidents or events.

Read as an engineering checklist, those four map onto things this series has already built. Map is the scope page. Measure is the evaluation and the monitoring. Manage is the rollback plan and the on-call rotation. Govern is the only one that cannot be delegated to an artefact, because it is the question of whether anybody is allowed to say no.

Use a framework to find your gaps. Do not use it to generate a workstream whose output is a compliance artefact nobody reads, because that consumes exactly the attention the real gaps needed.

The runbook, and the test that it is true

The last obligation is the one the plan phrased as documenting a system so somebody else can run it, and it is the easiest to verify.

One page. How to deploy it. How to roll it back. What each alert means and what to do first. Which upstream systems it depends on and who owns them. How to retrain it. What is known to be broken. Who to call.

Then test it the only way that works: hand it to somebody who has never touched the system and ask them to perform a rollback in a staging environment, unassisted, while the author stays silent. Whatever they get stuck on is the part of your runbook that was written for somebody who already knew the answer. Twenty minutes, once a quarter, and it is the difference between a document and a capability.

The same test applies to the whole system record. If a competent stranger can read what you have and answer the four questions at the top of this article, the governance is real. If they cannot, the process is producing paperwork, and the next questionnaire will be answered from memory again.

What the series was for

Seven articles, and one argument underneath them. A model that works in a notebook is the smallest part of a system that works in an organisation. The pipeline has to keep telling the truth. The experiment has to be reconstructible. The deployment has to be reversible. The failure has to be visible before a customer finds it. The economics have to close. And the whole thing has to be legible to somebody who was not there.

None of that is glamorous and none of it is optional. It is also, in every team that does it, the part that turns a promising model into a system that is still running in three years, which was the point of the exercise all along.

References

Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron and Parker Barnes, arXiv, 2020.
Model Cards for Model Reporting. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji and Timnit Gebru, arXiv, 2018.
Datasheets for Datasets. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume III and Kate Crawford, arXiv, 2018.
Artificial Intelligence Risk Management Framework (AI RMF 1.0): Core and Profiles. National Institute of Standards and Technology, 2023.