Artificial Intelligence1 min read

Introducing usage-based pricing for Amazon SageMaker AI bidirectional streaming

By · Published by Everything Blog

In short

The new billing capability addresses the technical gap where traditional HTTP response headers fail to track ongoing consumption in long-running bidirectional streams. To capture continuous usage, the system establishes a companion WebSocket connection specifically for transmitting metering telemetry alongside the primary data flow. Model servers report consumption using JSON text frames under the inference.count dimension, which can be sent incrementally or in periodic batches. Additionally, each companion connection enforces a rate limit of one message per second with frames capped at 512 bytes.

Key points

  • Customers will pay usage-based software fees separately from the charges for underlying h…: Customers will pay usage-based software fees separately from the charges for underlying hosting infrastructure.
  • Model servers must opt in using a specific header to initiate companion metering connecti…: Model servers must opt in using a specific header to initiate companion metering connections.
  • Providers can transmit single-record updates either as units are consumed or in periodic…: Providers can transmit single-record updates either as units are consumed or in periodic batches.

Sources checked by our analysis pipeline: 1

Artificial Intelligence

Introducing usage-based pricing for Amazon SageMaker AI bidirectional streaming

Usage-based metering for bidirectional streaming solves a software billing problem for real-time speech AI. Customers deploying streaming models on Amazon SageMaker AI bidirectional streaming pay for the endpoint infrastructure that hosts the model. Separately, model providers listing on AWS Marketplace need a way to charge for what customers consume during a live stream. Today, we’re announcing usage-based metering for Amazon SageMaker bidirectional (bidi) streaming inference, so model providers can report actual usage for the AWS Marketplace software charge. Amazon SageMaker AI endpoint infrastructure charges still apply.

Deepgram, the real-time speech AI company, is the launch partner for this capability, bringing usage-based Speech-to-Text (STT) and Text-to-Speech (TTS) billing to AWS Marketplace. This post builds on a series of innovations between Deepgram and AWS. First, we introduced bidirectional streaming for real-time inference on Amazon SageMaker AI, which supports long-lived bidirectional streams between clients and model servers. Then, Deepgram enhanced Amazon SageMaker AI support with AWS Identity and Access Management (AWS IAM) temporary delegation, simplifying secure authentication for enterprise deployments, and expanded…

Artificial Intelligence

More in Artificial Intelligence