A language model is a capable reader and an unreliable accountant. Any step that must produce the same output for the same input every time, and be explainable afterward, belongs in deterministic code. Any step that turns messy human input into structure is a reasonable job for a model, provided a deterministic check runs after it. The useful framing is not AI or no AI. It is which component is accountable for the output when a customer asks why.
How a model ends up doing a job it cannot hold
The demo works and the tenth case does not
Early testing uses clean examples, because those are easiest to find. The cases that decide whether a system survives are malformed, ambiguous, or open to two readings. A model handles those by producing something plausible, which is exactly the wrong behavior.
Nobody can answer the customer's why
A customer questions a number and somebody has to say where it came from. If the answer is that a model produced it, you have no answer, and you will either honor a number you cannot explain or refund it. Neither outcome is one you want to repeat.
The prompt becomes undocumented business logic
Rules get written into a paragraph of English inside a prompt, because that was easiest. That paragraph is now business logic with no version history, no test coverage, and no reviewer. It gets edited by whoever has access, and nobody knows when a rule changed.
The model changes underneath you
Versions get deprecated and replaced on the provider’s schedule, not yours. A change that improves general performance can still shift outputs on your edge cases. If a model sits on the decision path, that shift lands on customers before anybody internally notices.
Extraction is trusted without validation
A model reads a document and returns a number. It is in the right format and looks reasonable, so it gets used. Format is not correctness. Without a range check or a comparison against the record it belongs to, a confident misreading passes straight through.
Task by task: which half of the system should own it
Read the second column as a default rather than a law. What holds across every row is the reason in the third column, which is almost always about whether the answer must be reproducible and defensible.
| Task | Deterministic or model | Why | What breaks if you choose wrong |
|---|---|---|---|
| Calculating a fee, tax, or commission | Deterministic | The same inputs must produce the same number, and you must be able to show the arithmetic | Two customers quoted differently for identical work, with no way to explain either |
| Reading a scanned or emailed document for fields | Model, with validation after | Layout varies constantly, and this is pattern reading, which rigid code is bad at | Brittle templates that break the first time a sender changes their form |
| Deciding whether a submission is complete | Deterministic | Completeness is a checklist, and the checklist is knowable in advance | A submission accepted with a missing field that surfaces weeks later, downstream |
| Sorting free-text requests into categories | Model, behind a confidence gate | Human phrasing is unbounded, and keyword rules miss the long tail | Everything filed under other, or filed confidently in the wrong place |
| Approving something above a threshold | Deterministic | Authority is a policy somebody signed, not an inference | A system that approved something nobody gave it authority to approve |
| Drafting a first-pass reply or summary | Model | Language generation is the thing models are genuinely for | Staff writing the same message repeatedly, or text that reads like a form letter |
| Matching an incoming payment to an open invoice | Deterministic first, model on the remainder | Exact matches are arithmetic, and only the leftovers need judgment | Payments applied to the wrong account and discovered at reconciliation |
| Noticing that something looks unusual | Model, producing a flag rather than a verdict | Unusual is a comparison against a pattern, not a rule anyone can write | Legitimate work blocked, or the case you needed raised passing silently |
| Enforcing who may see which record | Deterministic | Permissions are a data question with a correct answer | Instructions hidden in user-supplied text talking a system into revealing something |
Assembling a workflow with both halves in their proper place
Draw the line where a number or a decision is produced
Walk the process and mark every point where something irreversible happens: a price set, an approval granted, a record created, a payment moved. Everything upstream of those marks is a candidate for a model. The marks themselves belong to code a person can read.
Give the model a schema to fill, not a question to answer
Asking for a structured object with named, typed fields produces something you can check. An open question produces prose that has to be parsed, and the parsing becomes its own failure source. Constrain the output shape and reject anything that does not match it.
Validate before anything acts on the output
Check ranges, check that referenced records exist, check that required fields are present, check that related values agree. This layer is unglamorous, and it is the difference between an extraction step you can run unattended and one that needs somebody watching.
Record what the model saw and what it returned
Store the input, the output, and the identity of the model and prompt version behind it, at the time it happened. When behavior shifts after a provider update, this is the only way to tell what changed and when. It is also how you answer a question about an old decision.
Keep the rules somewhere a non-programmer can read
Business rules belong in a table the person who owns the policy can review, not buried in code branches or prompt text. If the operations manager cannot read the current rules without asking a developer, they will drift from the policy they were meant to encode.
Adjacent decisions worth reading
- document processing and extraction workflows — The most common place a model earns its keep, and the place validation matters most.
- designing a fee calculation customers will trust — The deterministic side of this argument worked through in detail, using pricing as the example.
Is this a fit for your business?
A good fit when
- You are evaluating a proposal where a model sits somewhere on the decision path
- Documents or free-text requests arrive in volume and somebody reads every one
- A customer, regulator, or counterparty can ask you to justify an individual outcome
- An early pilot worked and you are deciding what to trust it with next
Probably not a fit when
- The process is fully structured and nothing arrives as prose or as a document
- Volume is low enough that a person reading each item is the cheaper design
- You are choosing a model provider, which is a different and later question
What to have ready
- A stack of real inputs including the ugly ones, not a curated sample
- The written policy for any step where a person currently makes a decision
- A clear statement of which outcomes are reversible and which are not
Questions we get asked
If we give a model enough past examples, can it price work?
It can produce numbers that look like your past numbers, which is not the same thing. Pricing must be reproducible on demand, defensible line by line, and stable when a rate changes on a known date. A model gives you none of those, and the failure mode is a wrong number nobody can trace.
Does setting temperature to zero make a model deterministic enough?
It removes one source of variation and leaves the ones that matter. The prompt changes, the retrieved context changes, the provider updates the model, and the same input produces a different output on a date you did not choose. Determinism here means reproducible next year, not twice this afternoon.
Where does a model belong in an approval workflow, if not the approval?
In front of it. A model is well suited to summarizing what is being approved, pulling relevant clauses out of an attached document, checking the request has what it needs, and flagging anything inconsistent with similar past requests. The approval itself stays with a rule and a named person.
Do we need to tell customers when a model was involved?
It depends on your industry and your contracts, and that is a question for counsel rather than your developer. From the engineering side, if a step is uncomfortable to disclose, the discomfort is usually a signal that it sits on the wrong side of the line described here.
Related
Tell us what the process looks like now and we will map what a system would need to do. No obligation, and you keep the map either way.