Most of the decisions around an LLM call are not about model selection. "Does this prompt contain a national ID", "which tier is this customer on", "can we write this to the cache", "which region is appropriate for this data class" — all of those are decisions. And they typically live inside application code, as independent if blocks scattered across it.
That scattering has two costs, and they arrive together.
The first cost: speed of change
When a regulatory interpretation changes, or a new data class is added, the change enters the release pipeline. Code change, review, test, deploy. We see this take two weeks in a good team. The bad part is not the duration — the bad part is paying that duration for a compliance change. Compliance decisions do not move at the same speed as product features; they usually move faster and less predictably.
The second cost: a decision you cannot explain
When an auditor asks why an answer came from a particular model, the answer has to be reconstructed from conditions spread across three different places in the code. In practice that means: it cannot be answered. The logs have the model name, maybe the rule name, but not the reasoning behind the decision.
Together these two lead to rules not being written at all. The team avoids the rule because it is risky, and the risk grows because there is no rule.
Moving the decision to one place
In Modelion these decisions are written in OPA/Rego, compiled to WASM, and evaluated inside the gateway on the request's hot path. The figures we measure are p50 154µs and p99 222µs — the decision does not show up in the request's total time.
But speed is not the point. The point is that the decision is made in one place, and that place is a rule set rather than code.
- name: national-id-pin-resident
when:
match: all
clauses:
- field: signals.pii.types
operator: containsAny
values: [tckn_tr]
then:
effect: route
route:
candidates: [gpt-4o-mini]
fallbackToOriginal: false
obligations:
redactTypes: detected
cache: { mode: off }
auditTags: [pii_detected]
When this rule goes into force, nothing changes in the application code. The application still sends a model name of combo/production-chat; policy decides which model that becomes.
Five axes, one decision
A decision is not a single-valued thing. Policy speaks on five axes at once:
- route — which model it should go to
- constrain — how the candidate pool narrows
- deny — whether the request is refused
- redact — what gets masked in the prompt
- cache — how the cache should behave
The merge rules for those axes matter: constraints intersect, obligations union. A constraint written at the platform baseline cannot be loosened by an organization rule beneath it. A team cannot add a foreign model to its own rule set — even if it is defined in the combo, it will not run.
That single sentence is what makes governance work in a multi-tenant product.
Shadow mode
Writing the rule is not enough; you need to know what it will do before it is enforced. In shadow mode the rule set is evaluated for every request but the result is not applied — only recorded. A week later you have this: how many requests agreed with current behaviour, how many diverged, and what the divergences were.
A rule set enforced without measuring divergence is a rule set learned in production.
The losing rules are recorded too
The decision trace holds more than the winning rule. Rules that matched and lost are listed as well. That looks like a detail at first, but it turns out to be the most useful thing in an audit: the answer to "did the compliance rule fire on this request" can be "yes, it matched but a more specific rule won" — and that is an entirely different answer from "no, it did not match".