Dense attention scales directly with context length, consuming the majority of prefill time at sequences reaching 128K tokens. Sparse attention decouples core attention from sequence length through a three-stage workflow involving key indexing, Top-k selection, and focused softmax attention. Despite these gains, indexing operations still evaluate every key and consume 47 percent of the prefill layer in models like DeepSeek V3.2. Hardware-tailored architectures mitigate this by adjusting head dimensions, compression ratios, and FP8 kernel granularity.
Key points
Transitioning to sparse attention eliminates core attention scaling issues by restricting…: Transitioning to sparse attention eliminates core attention scaling issues by restricting softmax operations to the highest-scoring keys.
Indexing mechanisms shift computational bottlenecks because they must still evaluate ever…: Indexing mechanisms shift computational bottlenecks because they must still evaluate every key across the full sequence context.
Optimizing sparse models on NVIDIA GPUs requires tuning indexer parameters and using FP8…: Optimizing sparse models on NVIDIA GPUs requires tuning indexer parameters and using FP8 kernels at token or small-block granularity.
Sources checked by our analysis pipeline: 1
Agentic workloads, long documents, and large codebases are pushing context lengths from tens of thousands of tokens to hundreds of thousands. At those lengths, attention becomes the most expensive part of a transformer layer. Dense attention scores every query against every key, so its cost per query grows with the context length N, and at 128K tokens it can take most of the prefill time in a layer.
The model-level solution is sparse attention, a transformer technique in which each query attends to only a small subset of the context, chosen per query, instead of all of it. This cuts both the compute and the memory traffic of attention on long context.
This post is a practical primer for model developers designing sparse attention that runs well on NVIDIA GPUs. It focuses on model-level sparse attention, where sparsity is trained into the model, rather than drop-in methods applied after training such as BLASST, MInference, or Quest. A production FP8 kernel is used to sweep the indexer’s design knobs (query heads, key heads, head dimension, compression ratio, and speculative-decoding depth), measure…
The London-based startup achieved a $1.5 billion valuation following its latest financing round co-led by General Atlantic and QED Investors. By utilizing open-source systems rather than frontier models, the firm aims to capture the financial services market by drastically reducing the operating expenses of routine labor.
The technology giant is turning its artificial intelligence platform into an active secretarial assistant capable of managing real-world errands autonomously. Users can delegate complex chores such as contacting local vendors, booking appointments, and completing transactions via connected payment systems. This transition shifts the tool from answering informational queries to performing direct actions on behalf of consumers.
The search giant revised its data management terms to hold onto sensitive interaction logs significantly longer than previously permitted. Users relying on history-off settings or ephemeral sessions will now have their conversational records preserved by the platform for several weeks. This shift marks a notable expansion in how long opt-out interactions remain stored on company infrastructure.