The engine provides an OpenAI-compatible interface running on a single trn2.48xlarge instance to serve large models like the 320-billion-parameter GLM-5.3-Flash. It pairs fused decode megakernels with a multi-tiered prefix cache that holds FP8 key-value pages and host-memory tiers to support 1-million-token contexts. Two tensor-parallel engines run across 32 logical NeuronCores managed by a session-affine router. During benchmark testing with 32 concurrent coding agents, the system sustained a processing rate of up to 3.57 million tokens per minute.
Key points
Optimized megakernels and tiered FP8 prefix caching allow single Trainium instances to ma…: Optimized megakernels and tiered FP8 prefix caching allow single Trainium instances to maintain context windows reaching one million tokens.
Deploying twin engines with tensor parallelism across 32 logical NeuronCores delivers hig…: Deploying twin engines with tensor parallelism across 32 logical NeuronCores delivers high-throughput capacity reaching 3.57 million tokens per minute.
Moving graph compilation entirely off the specialized serving hardware streamlines deploy…: Moving graph compilation entirely off the specialized serving hardware streamlines deployment while preserving OpenAI API compatibility for tools and reasoning.
Sources checked by our analysis pipeline: 1
Quickstart · Benchmarks · Docs · Kernels · Blog post · runanywhere.ai
TrnRT is an open inference engine for large language models on AWS Trainium. It serves an OpenAI-compatible API
from one trn2.48xlarge
, runs decode as NKI megakernels (about one kernel per layer half), and keeps a prefix cache
across device and host memory. GLM-5.3-Flash is the first supported model; more open models are on the way.
How we built it, with the AgentX results: Serving GLM-5.3-Flash on one Trainium2 box with NKI layer megakernels.
Feature | What it does |
---|---|
NKI megakernels | One fused kernel per layer half for decode: KDA attention, the fused DSA layer, mHC and the MoE FFN. |
1M-token context | Sparse attention (top 2,048) and linear attention, with fixed-size state. |
Prefix cache | A radix tree over FP8 KV pages, plus a host-memory tier for evicted pages and state checkpoints. |
Two engines per box | Each engine is tensor-parallel over 32 logical NeuronCores, behind a session-affine router. |
OpenAI API | Chat and completions, streaming, tool calls and reasoning…
The Swiss privacy provider is expanding its autonomous agent ecosystem to target computer automation without compromising user confidentiality. This cross-platform release directly challenges mainstream AI developers by coupling deep operating system integration with strict data protection guarantees. By leveraging localized European infrastructure, the rollout ensures consumer and corporate workflows remain fully exempt from surveillance laws and model retraining practices.
The technology giant has completed the purchase of the startup to integrate automated calendar and meeting management tools into its broader product ecosystem. This integration aims to enhance the company's autonomous digital assistant capabilities as competition intensifies across productivity software. The strategic transaction allows the parent company to immediately absorb specialized talent and proprietary algorithms tailored for modern enterprise workflows.
According to recent briefing reports, national preparedness has come under scrutiny after widespread technical failures disrupted corporate, financial, and government systems, including a classified National Security Agency communications network. JD Vance stated that the administration does not anticipate an imminent crisis, framing current defense readiness and domestic reinvestment as routine resilience measures. These interruptions have heightened scrutiny regarding the integration of third-party software and…