Artificial Intelligence1 min read

Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium

By · Published by Everything Blog

In short

The engine provides an OpenAI-compatible interface running on a single trn2.48xlarge instance to serve large models like the 320-billion-parameter GLM-5.3-Flash. It pairs fused decode megakernels with a multi-tiered prefix cache that holds FP8 key-value pages and host-memory tiers to support 1-million-token contexts. Two tensor-parallel engines run across 32 logical NeuronCores managed by a session-affine router. During benchmark testing with 32 concurrent coding agents, the system sustained a processing rate of up to 3.57 million tokens per minute.

Key points

  • Optimized megakernels and tiered FP8 prefix caching allow single Trainium instances to ma…: Optimized megakernels and tiered FP8 prefix caching allow single Trainium instances to maintain context windows reaching one million tokens.
  • Deploying twin engines with tensor parallelism across 32 logical NeuronCores delivers hig…: Deploying twin engines with tensor parallelism across 32 logical NeuronCores delivers high-throughput capacity reaching 3.57 million tokens per minute.
  • Moving graph compilation entirely off the specialized serving hardware streamlines deploy…: Moving graph compilation entirely off the specialized serving hardware streamlines deployment while preserving OpenAI API compatibility for tools and reasoning.

Sources checked by our analysis pipeline: 1

Quickstart · Benchmarks · Docs · Kernels · Blog post · runanywhere.ai

TrnRT is an open inference engine for large language models on AWS Trainium. It serves an OpenAI-compatible API

from one trn2.48xlarge

, runs decode as NKI megakernels (about one kernel per layer half), and keeps a prefix cache

across device and host memory. GLM-5.3-Flash is the first supported model; more open models are on the way.

How we built it, with the AgentX results: Serving GLM-5.3-Flash on one Trainium2 box with NKI layer megakernels.

Feature | What it does |

---|---|

NKI megakernels | One fused kernel per layer half for decode: KDA attention, the fused DSA layer, mHC and the MoE FFN. |

1M-token context | Sparse attention (top 2,048) and linear attention, with fixed-size state. |

Prefix cache | A radix tree over FP8 KV pages, plus a host-memory tier for evicted pages and state checkpoints. |

Two engines per box | Each engine is tensor-parallel over 32 logical NeuronCores, behind a session-affine router. |

OpenAI API | Chat and completions, streaming, tool calls and reasoning…

Artificial Intelligence