Let the machine be strict, and own every exception

A few months ago I watched a compliance engineer argue with a spreadsheet. A connector supplier had sent a certificate of conformance for a part. The certificate said the part met RoHS. It did not say what the part was made of. The engineer's question was whether that one page was enough, or whether she had to go back and ask for a full material declaration, which the supplier would take six weeks to produce and might refuse.

Nothing in the regulation answered her. The RoHS directive restricts ten substances at set concentrations in homogeneous materials. It does not tell you how much paper you need before you can believe a supplier. That judgment is the manufacturer's, and a European harmonized standard, EN IEC 63000, exists precisely because the directive leaves it open.¹

Here is the position I want to defend. In regulated work, the machine should produce the strictest defensible reading of the rule, and the business should own, record, and be able to defend every relaxation of it. Not the reverse. A system that starts lenient and asks a human to tighten it will drift toward whatever is cheapest. A system that starts strict and asks a human to loosen it drifts toward whatever the business can justify. Only one of those is auditable.

The problem, in the field's terms

Compliance is a mapping problem. On one side sit rules: a directive, a customer specification, an internal policy. On the other sits evidence: a supplier's declaration, a material declaration, a lab report. The job is to decide, for every part and every rule, whether the evidence satisfies the rule.

Both sides arrive as prose. Regulatory text is dense and hierarchical, with obligations nested inside conditions nested inside exceptions. A recent paper from a team at Vanguard describes the conversion of that text into machine-readable rules as "a costly, expert-intensive process," which matches my experience.² Evidence is worse. A certificate of conformance is, in the language of ISO/IEC 17050-1, a supplier's declaration of conformity: the supplier attesting, on its own authority, that the object meets specified requirements.³ A full material declaration is something else entirely. Under IPC-1752A, the electronics industry's declaration standard, a Class D declaration lists every substance in every homogeneous material of the part, exchanged as XML against a published schema.⁴ One document is a promise. The other is a bill of materials down to the chemistry.

The traditional answer has been people reading documents and filling in spreadsheets. It does not scale, and it does not version. When the rule changes, nobody re-reads last year's certificates. CUBE's 2025 survey of more than 2,000 compliance, risk, and legal leaders found that 74 percent of firms take more than a year to implement a new regulation.⁵ A year is roughly how often the substance lists move.

The mechanism

What a language model changes is the cost of the two readings. It can read a regulation and emit typed rule units: action, condition, constraint, exception, penalty. It can read a certificate and emit typed facts: part number, standard claimed, substances declared, date, signatory. The Vanguard team's pipeline, De Jure, does the first of these with no human annotation and no domain-specific prompting, scoring between 4.74 and 4.85 out of 5 on an SEC corpus under a multi-criteria judge, and holding between 4.71 and 4.85 when pointed at HIPAA and the EU AI Act without modification.² Their hardest stage, by their own account, was the fine-grained one: recovering conditions, exceptions, and nested qualifiers. That is exactly the stage where compliance lives.

Two engineering facts constrain how you can use this.

First, the model is not deterministic, even when you tell it to be. Atil and colleagues ran five models configured for deterministic output across eight tasks, ten runs each, and saw accuracy vary by up to 15 percent between runs, with a best-to-worst gap of up to 70 percent. None of the five produced identical output strings across all tasks.⁶ Horace He at Thinking Machines showed why: the kernels in production inference servers are not batch-invariant, so a request's numerical result depends on how many other requests share its batch. At temperature zero, asking one model the same question 1,000 times produced 80 distinct completions, all identical for 102 tokens and diverging at the 103rd.⁷ His batch-invariant kernels fixed it at roughly 1.6 to 2 times the latency. Most people are not running those kernels.

Second, the fix everyone reaches for, constraining the output to a schema, guarantees form and nothing else. OpenAI's own documentation is careful: Structured Outputs ensures the model "will always generate responses that adhere to your supplied JSON Schema." It does not claim the content is correct.⁸ And forcing structure has a cost. Tam and colleagues measured what happens when you make models answer reasoning tasks in JSON. Claude 3 Haiku fell from 86.5 percent to 23.4 percent on GSM8K under JSON mode; GPT-3.5-turbo fell from 76.6 to 49.3 percent. The same paper found the opposite for classification, where the restricted answer space helped, and recommended a two-stage approach: reason in natural language first, then convert.⁹

Put those together and the architecture draws itself. The probabilistic component, the model, does extraction and only extraction, in two passes, with its output landing in a schema a person can read and correct. The decision, whether this evidence satisfies this rule, is a deterministic function over the extracted rule set and the extracted facts. Same inputs, same verdict, every time, with no model in the loop. The policy world has a name for this separation. The Open Policy Agent project, a graduated CNCF project, calls it decoupling: policy is "a set of rules that governs the behavior of a software service," specified declaratively, updated at any time without recompiling or redeploying, and enforced automatically.¹⁰ What a language model adds is that the rules can now be extracted from the regulation rather than hand-written.

That is what "deterministic AI" ought to mean in this domain. You do not have a deterministic model. You can have a deterministic decision layer, fed by a probabilistic reader whose every output is inspectable before it becomes a rule.

Where the strictness comes from

Rule extraction leaves a residue. Every regulation has clauses whose evidentiary requirement is not stated. RoHS never says whether a certificate of conformance is sufficient for a connector. EN IEC 63000 tells the manufacturer to take a documentary, risk-based approach, and names three kinds of evidence: supplier declarations or contractual agreements, material declarations, and analytical testing.^1,11 Which one you need for which part is your call.

The machine has to fill that gap, because a rule with an unspecified evidence requirement cannot be evaluated. It can fill it with the least demanding evidence that could plausibly satisfy the clause, or with the most demanding. I am arguing for the most demanding, for two reasons that have nothing to do with caution as a virtue.

The first is about who owns the error. If the default is lenient and it turns out to be wrong, the wrongness was decided by a model, nobody signed it, and there is no record of the decision because no decision was visibly made. If the default is strict and a person relaxes it, there is a person, a date, and a reason. Regulators are already writing this asymmetry into law. Article 14 of the EU AI Act, for high-risk systems, requires that the people overseeing a system be able "to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output."¹² A compliance tool is not itself a high-risk system under the Act, but the design principle is the right one for anything that produces a verdict a company will later have to stand behind: the override is the human act, and it has to be visible.

The second is that the relaxation is where the business knowledge actually lives. The regulation does not say whether a given supplier can be trusted on a declaration alone. Your history with that supplier says so, along with your incoming inspection data and the material's known risk. That knowledge belongs in a configuration layer the business writes, versions, and can explain, rather than in a model's guess about what "sufficient" means.

A worked example

At Certivo we run it this way. The extraction pass reads a regulation, a customer standard, or an internal policy and produces the rule set, with each rule's evidence requirement set to the strictest reading the text supports. For a RoHS obligation on a purchased part, that baseline is a full material declaration and a certificate of conformance, because a declaration proves composition and a certificate proves the supplier is willing to attest to it, and the text does not say either alone is enough.

The business then writes its logic on top. For a supplier with three years of clean declarations, on a material class with no history of restricted substances, a certificate of conformance alone satisfies the rule. That relaxation is a configuration entry: which rule, which scope, who approved it, when, and why. On the evidence side the same reader extracts the certificate's facts and maps them to the rule. The decision is then a lookup: does this part, from this supplier, under this configured rule, have the evidence the rule now demands? If the supplier changes, the configuration does not travel with the part, and the baseline comes back.

The engineer with the spreadsheet is still in the loop. She is just no longer the one deciding what the regulation means. She is deciding, on the record, when the company knows enough to accept less than the regulation could be read to demand.

The strongest objection

The objection is that "a human owns the relaxation" is a slogan, and that in practice the human clicks approve. Parasuraman and Manzey's 2010 review in Human Factors is the standard reference. They define automation bias as the tendency to make "both omission and commission errors when decision aids are imperfect," and complacency as what happens "under conditions of multiple-task load, when manual tasks compete with the automated task for the operator's attention." Neither is fixed by expertise or by practice.¹³ The EU AI Act names the same failure in Article 14(4)(b), requiring that overseers stay aware of "the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)."¹²

They are right, and the design has to assume it. The point of putting strictness in the default is that it sets the direction the bias runs. If accepting the machine's default is the lazy path, then the lazy path is the conservative one. Where automation bias will bite is the other way: a relaxation, once approved, tends to stay approved, and reviewers of a long-standing exception will wave it through. So exceptions need expiry dates and re-approval, and the rate of relaxation per reviewer needs to be a metric someone looks at. That is a real cost, and so is the over-collection the strict default causes before the exceptions are written. I would rather pay it than argue with a regulator about a decision no one made.

What this means if you run the engineering

Three things follow for a CTO or VP of engineering putting one of these systems in.

Insist that the decision layer is a pure function, and test it like one. Replay last quarter's rule set and facts and confirm the verdicts are byte-identical. If a model call sits anywhere in that path, you have built a system that can give two answers to an auditor.

Treat the extracted rule set as a versioned artifact with a review gate, the way you treat a schema migration. The model proposes; a person with domain authority merges. The nondeterminism findings are the argument for the gate; the De Jure results are the argument that the proposals will be good enough to make review worth doing.

Make relaxation a first-class object in the data model, with an owner, a scope, a reason, and a lifetime. If you cannot list every place your system is accepting less than the strictest reading, and who agreed to it, you have a spreadsheet with better handwriting.

The rules will keep changing. Substances get added, thresholds move, a customer sends a new specification. The machine can re-extract and re-run every decision in an afternoon. What it cannot do, and should not be asked to do, is decide how much you trust your suppliers. That was always your job. The difference is that you can now see, in one table, every time you exercised it.

— Kunal