Artificial Intelligence1 min read

How Airbnb uses cost per task to operate coding agents on Amazon Bedrock

By · Published by Everything Blog

In short

The travel platform routes coding tasks through its AirChat tool and a central model gateway to Amazon Bedrock, which recently absorbed a shift of about two-thirds of requests toward newer model offerings. Pre-rollout testing evaluates the list-price token expenditure across allowed reasoning effort levels on internal tasks. After launch, Amazon CloudWatch tracks runtime metrics such as throttling, latency, and cached input rates to verify production efficiency. By comparing live aggregate gateway logs to original benchmark scores, administrators dynamically adjust system defaults and reasoning ceilings.

Key points

  • Custom internal task evaluations determine default coding models and maximum reasoning ef…: Custom internal task evaluations determine default coding models and maximum reasoning effort caps instead of public industry benchmarks.
  • Production performance is continuously verified using Amazon CloudWatch metrics that trac…: Production performance is continuously verified using Amazon CloudWatch metrics that track token consumption, latency, throttling, and cached input rates.
  • Discrepancies between pre-rollout benchmarks and live operational telemetry trigger admin…: Discrepancies between pre-rollout benchmarks and live operational telemetry trigger administrative adjustments to platform defaults and reasoning ceilings.

Sources checked by our analysis pipeline: 1

Artificial Intelligence

How Airbnb uses cost per task to operate coding agents on Amazon Bedrock

Amazon Bedrock provides access to foundation models from multiple providers through a common API, with AWS managing the model infrastructure. It integrates with AWS access controls and monitoring, and processes prompts, code, and model responses within AWS.

As coding agents move into production, changes in models and usage can affect latency, capacity, and cost. Amazon Bedrock runtime metrics in Amazon CloudWatch help teams investigate latency increases, throttling, and changes in token consumption. Evaluations, or evals, complement these operational metrics by assessing the quality of an agent’s work on a fixed set of representative tasks.

In this post, we show how Airbnb evaluates models for coding agents and monitors their use on Amazon Bedrock. We explain how cost per task guides model selection and how production data helps us revisit those choices as usage changes. We also describe the runtime signals used to investigate latency, throttling, and spend.

At Airbnb, thousands of engineers and engineering-adjacent people use AirChat, our internal tool for running coding agents, every…

Artificial Intelligence