A new policy behaves correctly on your sample requests. The next question is what it will change when it encounters real traffic.
Shadow mode lets you investigate that question while the existing decision continues to govern the request. A candidate policy is evaluated alongside it. The difference between those decisions becomes evidence for the rollout decision.
Describe the expected difference first
When you add a redaction rule, identical behaviour is not the objective. Some requests should acquire a new obligation. Alongside "are these decisions equal?", ask "did the difference occur on the requests we intended to change?"
Write a short acceptance table before collecting results:
| Request group | Expected change | Evidence to inspect |
|---|---|---|
| Target data type detected | Add a redaction obligation | Detection signal and obligation |
| Target data type absent | Keep the existing route | Existing and candidate decisions |
| Scanner result unavailable | Apply the defined failure behaviour | Signal availability |
This is a test plan, not a product configuration. It helps reviewers agree on what a result means.
Separate divergences by impact
A different model and a refused request have different consequences. Redaction, disabling the cache and narrowing the candidate set should also be reviewed independently.
Modelion decisions operate on several axes. Looking only at the final model name conceals some of those changes. Two decisions can choose the same model while changing the text sent upstream or the obligations attached to execution.
Start the review queue with unexpected refusals and constraint violations, then route changes, then other obligations. Record how each difference relates to the intended behaviour. This gives the team something more useful than a single pass percentage.
Know which traffic you have not observed
A test window without overnight traffic provides weak evidence for an after-hours rule. An infrequently used model, a new prompt version or a small team can be underrepresented in the same way.
Break down the aggregate by organization, virtual key, prompt version and request type. Mark groups with too few examples explicitly. Calling them healthy would confuse missing evidence with evidence of correctness.
A deliberate sample request can help exercise an uncovered condition, but keep those results distinguishable from observations of normal traffic. They answer related, different questions.
Measure the cost of evaluation
Logical correctness does not tell you how expensive a policy is to run. Two rules producing the same result can have different evaluation costs. Modelion's publish-time cost check and shadow observations on real traffic therefore serve different purposes.
OPA's performance guidance also emphasizes benchmarking policy evaluation with representative inputs. OPA policy performance documentation.
Write the rollback condition before rollout
Identify the version to enforce, the reviewers responsible for divergences, and the conditions for returning to the previous version. Use observable conditions: unexpected refusals, an uncovered critical request group, or an unexplained increase in evaluation cost.
Shadow mode is a bounded experiment. Its value comes from showing precisely what a new policy will change, and where the evidence is still incomplete.
