To reduce generative AI costs, measure how many requests can be handled by rules, what each remaining path costs, and whether it meets your quality and latency targets. AWS recommends using deterministic code for checkable rules, constrained decisions for fixed choices, and right-sized models for open-ended work (AWS’s workflow guidance).
That gives a team a practical way to compare its own workflow instead of relying on a generic savings percentage. The key is to count all the paths a request can take—not just the model call.
Compare costs using your workload
For a classification-only task, compare a model-only route with a rule-first route:
N × C_M versus N × C_R + N × (1 − r) × C_D
Here, N is the number of requests, r is the measured share that rules resolve, C_M is the cost of one model classification, C_R is the cost of one rule execution, and C_D is the cost of one constrained decision for an unresolved request.
The formula assumes both paths handle the same task to an acceptable standard. If a constrained decision can escalate cases to a generative model, include the cost of those calls too. Add differences in infrastructure, evaluation and operations separately; leaving them out can make one route look cheaper than it is. Compare cost per successful outcome, not merely cost per request.
AWS’s example assumes rules resolve 95% of requests, leaving one in 20 to reach a model. That is illustrative arithmetic, not a measured result. Your own value for r depends on the rules and requests in your workflow.
Count every route a request can take, then compare the cost of successful outcomes against the same quality and latency requirements.
Match the execution path to the task
The best candidate for a rule is a step with a result that can be checked reliably: a required format, schema, keyword or numeric range. A fixed-choice decision is different: it selects from a defined set, but may need more than a simple rule. Open-ended writing or interpretation still calls for a generative model.
| Workflow task | Execution path | Why it fits | Practical consideration |
| Check an order format or a reviewed tracking-number pattern | Deterministic code | The expected result can be checked against a known rule | Keep authorization and range checks in code; don’t execute arbitrary code from configuration files |
| Route an intent that rules cannot resolve | Fast path, then a constrained decision | A defined set of destinations makes the decision bounded | Test uncertainty and escalation signals against labeled examples |
| Choose a support queue | Constrained decision | The answer is one option from a fixed set | Set an option such as “unsure” when the decision method supports it |
| Draft a customer reply | Right-sized generative model | Writing a response is open-ended work | Choose a model that meets the task’s quality floor and latency target |
AWS lists structured outputs, embedding similarity, experimental Strands Decider and a trained classifier as options for fixed-choice tasks. Treat confidence thresholds and escalation rules as decisions to evaluate against labeled data, not as settings that are automatically reliable for every workflow.
Look beyond model calls
Routing is one part of the bill. Cloud With Raj also discusses managing prompt context, batching related tasks, semantic caching, cleaning up retrieval-augmented generation (RAG) documents, choosing vector-store and infrastructure options, and model distillation. These are separate areas to assess in your own system; they don’t provide a universal savings rate.
For example, a team might first inspect whether a request carries more retrieved context than its task needs, then evaluate whether repeated requests can use a cache. Those changes affect different parts of a workflow, so measure them separately from the share of requests resolved by rules.
Evaluate quality, latency and compliance together
Before changing a route or model, set a quality floor, a cost ceiling, a latency target and the applicable compliance constraints. Test alternatives on labeled examples from the task, track cost per successful outcome, and keep a rollback path. Continue monitoring for drift: a rule or decision threshold that worked on earlier requests may need reassessment as the workload changes.
A cheaper route is useful only when it still meets those requirements. The measured request mix, per-path costs and evaluation results are what make the comparison specific to your workflow.