To reduce generative AI costs, measure how many requests can be handled by rules, what each remaining path costs, and whether it meets your quality and latency targets. AWS recommends using deterministic code for checkable rules, constrained decisions for fixed choices, and right-sized models for open-ended work (AWS’s workflow guidance).

That gives a team a practical way to compare its own workflow instead of relying on a generic savings percentage. The key is to count all the paths a request can take—not just the model call.

Compare costs using your workload

For a classification-only task, compare a model-only route with a rule-first route:

N × C_M versus N × C_R + N × (1 − r) × C_D

Here, N is the number of requests, r is the measured share that rules resolve, C_M is the cost of one model classification, C_R is the cost of one rule execution, and C_D is the cost of one constrained decision for an unresolved request.

The formula assumes both paths handle the same task to an acceptable standard. If a constrained decision can escalate cases to a generative model, include the cost of those calls too. Add differences in infrastructure, evaluation and operations separately; leaving them out can make one route look cheaper than it is. Compare cost per successful outcome, not merely cost per request.

AWS’s example assumes rules resolve 95% of requests, leaving one in 20 to reach a model. That is illustrative arithmetic, not a measured result. Your own value for r depends on the rules and requests in your workflow.

Count every route a request can take, then compare the cost of successful outcomes against the same quality and latency requirements.

Match the execution path to the task

The best candidate for a rule is a step with a result that can be checked reliably: a required format, schema, keyword or numeric range. A fixed-choice decision is different: it selects from a defined set, but may need more than a simple rule. Open-ended writing or interpretation still calls for a generative model.

Workflow taskExecution pathWhy it fitsPractical consideration
Check an order format or a reviewed tracking-number patternDeterministic codeThe expected result can be checked against a known ruleKeep authorization and range checks in code; don’t execute arbitrary code from configuration files
Route an intent that rules cannot resolveFast path, then a constrained decisionA defined set of destinations makes the decision boundedTest uncertainty and escalation signals against labeled examples
Choose a support queueConstrained decisionThe answer is one option from a fixed setSet an option such as “unsure” when the decision method supports it
Draft a customer replyRight-sized generative modelWriting a response is open-ended workChoose a model that meets the task’s quality floor and latency target

AWS lists structured outputs, embedding similarity, experimental Strands Decider and a trained classifier as options for fixed-choice tasks. Treat confidence thresholds and escalation rules as decisions to evaluate against labeled data, not as settings that are automatically reliable for every workflow.

Look beyond model calls

Routing is one part of the bill. Cloud With Raj also discusses managing prompt context, batching related tasks, semantic caching, cleaning up retrieval-augmented generation (RAG) documents, choosing vector-store and infrastructure options, and model distillation. These are separate areas to assess in your own system; they don’t provide a universal savings rate.

For example, a team might first inspect whether a request carries more retrieved context than its task needs, then evaluate whether repeated requests can use a cache. Those changes affect different parts of a workflow, so measure them separately from the share of requests resolved by rules.

Evaluate quality, latency and compliance together

Before changing a route or model, set a quality floor, a cost ceiling, a latency target and the applicable compliance constraints. Test alternatives on labeled examples from the task, track cost per successful outcome, and keep a rollback path. Continue monitoring for drift: a rule or decision threshold that worked on earlier requests may need reassessment as the workload changes.

A cheaper route is useful only when it still meets those requirements. The measured request mix, per-path costs and evaluation results are what make the comparison specific to your workflow.