Build compliance software in house only when nobody outside audits the output

Every build-or-buy review I have sat through for compliance tooling has started with the wrong question. The engineering lead estimates the build. The finance lead prices the subscription. Somebody opens a spreadsheet, pastes a supplier declaration into a chat window, gets a confident answer in four seconds, and the room quietly concludes that the whole category might be a solved problem.

Two years later a customer's auditor asks a different question: which document supported this determination, which version of the rule applied on that date, and who approved it. Nobody in the original meeting priced that question. It is the only one that matters.

My claim: the build-or-buy decision for a compliance system of record turns on whether an outsider can reconstruct a determination years after it was made. Reconstruction needs deterministic rules, source binding and a rule set someone maintains continuously. Those three things are a product, not a project, and no general-purpose model supplies them. The corollary is the title. Build in house when nobody outside the company will ever audit the output. Otherwise the question is which product, not whether.

The auditor's question, in the auditor's terms

A compliance system of record is the system whose entries are treated as the authoritative account of what the company decided and why. The term matters because it sets the standard the system is held to. That standard is written down.

ISO 19011, the guideline every management-system auditor is trained on, defines audit evidence as "records, statements of fact or other information, which are relevant to the audit criteria and verifiable," and audit criteria as the "set of requirements used as a reference against which objective evidence is compared."¹ Verifiable is the load-bearing word. A statement the auditor cannot independently confirm is not evidence; it is a claim.

The retention horizon is also written down. Under the EU RoHS Directive a manufacturer must "keep the technical documentation and the EU declaration of conformity for 10 years after the EEE has been placed on the market," and must "keep a register of non-conforming EEE and product recalls."² Ten years is longer than most internal tools survive their original author, and far longer than the memory of a chat session.

So the specification is set by the field, not by the vendor or the engineering team: for each determination, hold the source document, the rule and its version, the logic that connected them, and the approver, in a form a third party can check a decade on. Any approach that cannot meet that specification is not a compliance system of record, however good its first answer looks.

Why the model cannot be the record

The tempting shortcut is to let a large language model read the declaration and say whether the part complies. It fails the specification on two independent counts, and both are measured.

The first is reproducibility. Practitioners set the sampling temperature to zero and assume the output is now a deterministic function of the input. It is not. Atil and colleagues ran five models, including GPT-4o, ten times each at temperature zero with a fixed seed across eight benchmark tasks and found "accuracy variations up to 15% across naturally occurring runs," with the share of runs returning byte-identical output ranging from near zero to about 99 percent depending on model and task.³ Horace He at Thinking Machines traced the cause in September 2025: production inference servers batch requests together, the batch size depends on other users' load, and most GPU kernels are not batch-invariant, so floating-point additions happen in a different order and "(a + b) + c ≠ a + (b + c)." His test sent the same prompt a thousand times at temperature zero and got 80 distinct completions.⁴ OpenAI's own documentation says the seed parameter is a "best effort," that "determinism is not guaranteed," and that responses can differ "due to the inherent non-determinism of our models."⁵ The property auditors call reproducibility is the property hosted models explicitly disclaim.

The second is grounding. A generative model answers from its weights, and the weights do not distinguish a rule it has seen from one it has interpolated. Dahl, Magesh, Suzgun and Ho measured this in the legal domain and found hallucinations, "textual output that is not consistent with legal facts," in 58 percent of responses from GPT-4 and 88 percent from Llama 2 on verifiable questions about federal cases, and that the models "often fail to correct a user's incorrect legal assumptions."⁶ Regulatory text is legal text. A model that returns a substance threshold with the same fluency whether it read it or invented it is, in ISO 19011's terms, producing statements of fact that are not verifiable.

Neither finding says models are useless in compliance. They say where a model may sit in the architecture. It can read, extract and classify, and each of those outputs can be checked against a source. It cannot be the thing that decides, and it cannot be the record.

What a system of record is made of

Three components carry the specification. Each is a well-understood piece of engineering; the difficulty is keeping all three alive at once.

The first is a deterministic rule engine: the restriction, threshold, exemption and scope conditions encoded as explicit logic that, given the same extracted facts and the same rule version, returns the same determination every time. This is the part a model cannot replace, because determinism is the definition.

The second is source binding, which is the provenance problem under a compliance name. The W3C's PROV data model defines provenance as "a record that describes the people, institutions, entities, and activities involved in producing, influencing, or delivering a piece of data," and defines derivation as "the construction of a new entity based on a pre-existing entity."⁷ In practice: every extracted value points at a document, a page and a region; every determination points at the values and the rule version it was derived from; every approval points at the determination. When the auditor asks which document supported the call, the answer is a pointer, not a search.

The third is rule versioning with immutable history. The rule that applied on the date the product shipped is what the auditor will test against, so old versions are never edited, only superseded, and each determination stays bound to the version in force when it was made.

Figure 1 puts the three common approaches against those properties. The spreadsheet-and-model path fails the first three outright: an editable cell is not an immutable record, a chat transcript is not source binding, and the model is not deterministic. The custom build can supply all three, and usually does, after the first audit finds the gap. The question for the build is not capability. It is the fourth row.

The rule set moves, and maintenance is most of the cost

The regulations a compliance system encodes are not a fixed input. The EU's Candidate List of substances of very high concern was first published on 28 October 2008 with 15 entries; in February 2026 it reached 253, when n-hexane and bisphenol AF were added.⁸ Between those dates it was amended 35 times, about twice a year. Each addition changes the answer for every article containing that substance above 0.1 percent by weight, and it re-opens the supplier communication and database notification duties that hang off the list. Multiply by every regulation in scope and every jurisdiction, and the rule set is a stream, not a file.

Software economics say what that stream costs. Robert Glass's summary of the empirical record, still cited a quarter-century on, is that "maintenance typically consumes about 40 to 80 percent (60 percent average) of software costs," and that about 60 percent of that maintenance is enhancement rather than bug-fixing.⁹ A rule set that changes on the regulator's calendar is pure enhancement load, forever. The 2012 McKinsey and Oxford study of more than 5,400 IT projects found that large ones "run 45 percent over budget and 7 percent over time, while delivering 56 percent less value than predicted," and that "software projects run the highest risk."¹⁰ That figure is about the build. The maintenance figure is about the decade after.

Martin Fowler's 2010 test decides which side of the line a function sits on: "it's all about whether the underlying business function is a differentiator or not," and "for a utility function you buy the package and adjust your business process to match the software."¹¹ For most manufacturers, substance compliance is a condition of selling, not a reason customers choose them. It is a utility with the maintenance profile of a strategic system, which is the worst combination to build.

What it looks like when it holds

A compliance team I work with runs the loop this way. Supplier documents arrive in whatever format the supplier uses. A model extracts substance data and confidence scores; nothing it produces is a determination. The rule engine matches the extracted values against the versioned rule set and returns pass, fail or insufficient evidence for each part and regulation. Items with low confidence, conflicting declarations or missing coverage route to a compliance engineer, who reviews the exception and approves or overrides, with the override itself recorded against the rule version. When a regulation changes, the new version is added, the whole portfolio is re-screened against it, and only the parts whose status changed come back to a human.

Two properties follow. The team's review effort scales with exceptions, not with parts, so it survives the deadline week that turns full manual review into spot checks. And when the customer's auditor asked, last quarter, which declaration supported a determination made under a rule that has since been superseded, the answer was a link.

The strongest case against

McKinsey's State of AI in 2026, published in August from a survey of 1,719 respondents, reports that "nearly one-third of respondents (32 percent) report that their organizations have decided against purchasing at least one software product or feature because they were able to build the functionality in-house using agentic coding tools," and that nearly half of AI high performers did so.¹² If coding agents have collapsed the build cost, the maintenance objection weakens too: the same agents can encode each regulatory change.

That is the right objection, and it is partly correct. Agents do lower the cost of writing the rule engine and the provenance layer, and any vendor whose value is "we wrote the code" should be worried. But the survey measures a decision, not an outcome. Nothing in it speaks to whether the systems built survived their second audit. And the two things the build cannot get from an agent are exactly the two the specification demands: a regulatory content feed that someone reads, interprets and re-encodes on the day the list changes, and an extraction layer tuned on the document formats that suppliers across an industry actually send, which no single company's inbox contains. Cheaper code sharpens the question; it does not answer it.

Decisions for the CTO

Three follow from the argument.

Decide on the audit, not the demo. Before comparing any two options, write the auditor's question for your product line, with the retention horizon your regulations impose, and score each option on whether it can answer it in year eight. Anything that cannot is not in the comparison.

Put the model where it can be checked. Extraction, classification and triage are model work; determination and record are not. An architecture that lets a generative output become the record has already failed the specification, whoever built it.

Price the stream. If you build, budget the rule set as a permanent enhancement backlog against the published cadence of every regulation in scope, and name the person who owns each one. If that name is a compliance analyst who also reviews parts, you have built the spreadsheet with extra steps.

Figure 3 is the decision as a plot. In the bottom-left, a stable, narrow rule set and no external auditor, the build is the right call and a spreadsheet may even do. Everywhere else the outsider's question is coming, and the only systems that answer it are the ones maintained as products.

I built one, and I still think most companies should not. The reason is not that the code is hard. It is that the rule set never stops moving, and the auditor never stops asking.

— Kunal