Airbnb measures a coding agent’s cost per task by adding the list-price cost of all tokens used across the requests needed to complete it, then pairing that figure with a grader’s quality score. The company also checks real production usage before revisiting model choices: a prepared-task result alone does not capture how often people make requests or how much context those requests consume. In a case study published by AWS on November 18, 2026, Airbnb described this approach for its internal AirChat tools, which route requests through an LLM gateway to Amazon Bedrock. Airbnb’s account says thousands of engineers and engineering-adjacent staff use AirChat weekly.
What Airbnb counts as the cost of a coding task
Airbnb counts all requests needed to finish a task, including their uncached input, cache reads, cache writes and output tokens. Each category has its own price. The company says its average task cost includes low-scoring attempts, so the figure reflects the cost of the evaluation runs rather than only the successful-looking results.
That makes the measure different from the cost of a single prompt or a model’s score on one benchmark. A task may take several requests, and its total depends on both the token mix and the number of attempts. Airbnb compares this cost with a grader score for each model, reasoning-effort setting and agent client it evaluates.
How Airbnb evaluates coding agents
Airbnb describes seven suites of representative engineering tasks. Each task starts from a repository state and an engineer-style prompt, then a trusted grader assesses the result. The tasks draw on real code changes, review comments and decisions that may not appear in the original ticket. Airbnb says an evaluation can produce initial results within hours, with suite-wide scores and costs by the following morning once tasks are prepared.
The company’s rubric scores eight dimensions from 0 to 3, with different weights: correctness (25), verification (15), autonomy (15), design and workload fit (10), judgment (10), evidence and reporting (10), readability (10), and operational discipline (5). If autonomy scores 0 or 0.5, the total is capped at 25 out of 100. That cap makes the ability to work independently a constraint on the overall score, not just one more quality category.
Airbnb’s example of reasoning effort shows the trade-off a paired score-and-cost measure can expose. On one unnamed model, moving from low to high effort raised the evaluation score by 0.8 points while making cost per task 2.5 times as high. Airbnb says it uses such results to set model defaults and effort limits.
Compare the cost of finishing the task with the quality of the result; a cheaper request is not automatically a cheaper completed task.
What Airbnb observed in production
Airbnb says model A’s requests cost about three times as much as requests on its predecessor, while average users sent about three times as many requests. Average weekly spending per user was several times higher. The company attributed most of model A’s request cost to cached context being read again during long, self-directed sessions; output tokens accounted for about one tenth of the request cost.
The distinction matters because a fixed evaluation controls the task, while production reflects people’s actual request patterns and the context carried through longer sessions. Airbnb says a newer, unnamed model scored higher than model A on the same tasks and with the same grader at about one sixth of its cost per task. After moving model A’s heavy users to the newer model, Airbnb reported that their spending on their main model in the first full week fell to about one seventh of its previous level, with roughly the same number of requests. The reported migration result concerns that user group and week.
Why billed tokens and quota use can differ
Token billing and quota admission count usage differently. Airbnb describes quota admission as reserving uncached input, cache writes and the maximum output allowance, or max_tokens; cache reads do not count toward that reservation. Output-token quota burndown can also vary by model, so a billed-token dashboard and an admission estimate may diverge.
For example, Airbnb’s account calculates an initial reservation of 4,696 tokens from 600 input tokens plus a max_tokens setting of 4,096. Lowering that setting to 1,024 reduces the reservation to 1,624 tokens, about 65% less. If the actual input and output remain unchanged, the final quota consumption remains unchanged too; the smaller setting changes the amount reserved up front.
Airbnb also describes monitoring latency, time to first token, output-token counts, throttling, estimated quota use and token categories in Amazon CloudWatch. The company records model, client, team and user through its gateway for aggregate cost and capacity planning.
Operational controls around coding agents
Airbnb recommends running agents in isolated workspaces, testing generated changes before merging them, and defining which actions an agent may take and where human review is required. Those controls address what an agent can change; cost and quota monitoring address how it behaves while doing the work.
Airbnb notes that invocation logging can contain full prompts, responses and caller identity, and recommends restricting access and setting retention. It also says cost-allocation tags do not apply retroactively to earlier costs.