The architecture integrates horizontal states that persist over time to condition attention, key-value cache writes, multi-layer perceptron processing, and decoding. During evaluations on 20 billion FineWeb-Edu tokens, a 406-million-parameter implementation dropped validation perplexity from 15.41 down to 14.40 compared to a standard baseline. It also reduced key-value cache size by 45% because the design achieved these results using only 22 layers compared to the baseline's 40 layers. Furthermore, combining the setup with hyper-connections and input residuals drove perplexity down even further to 13.61.
Key points
Horizontal recurrent latent streams run alongside vertical streams to condition core comp…: Horizontal recurrent latent streams run alongside vertical streams to condition core components like attention, key-value caching, and decoding.
The architectural framework successfully enhances non-Transformer models including Gated…: The architectural framework successfully enhances non-Transformer models including Gated DeltaNet, Mamba-2, Triadic GDN, and hybrid variants.
Lattice configurations outperformed pure recurrent baselines that maintained four times l…: Lattice configurations outperformed pure recurrent baselines that maintained four times larger native states, confirming persistent per-layer states as a viable scaling method.
Sources checked by our analysis pipeline: 1
1MIT2Adobe
Abstract
Language models typically increase per-token computation through deeper or looped layers and latent reasoning steps, while richer residual connections improve information flow across layers. We instead increase computational capacity: the number of active computational states a model maintains. We introduce the Lattice of Latents, a set of per-layer horizontal latent streams that run recurrently along time, alongside the vertical residual stream that flows through depth. Like hyper-connections, mHC, and attention or value residuals, the Lattice enhances the residual stream, but rather than adding or rerouting residual pathways through depth, it provides companion computational states that persist across tokens. Unlike hybrid Transformer–SSM architectures, whose recurrent states primarily serve as internal memory for sequence mixing, these latents directly condition the vertical stream’s attention, KV-cache writes, MLP computation, and decoding. On Transformers trained on 20B FineWeb-Edu tokens, a 406M-parameter Lattice model lowers validation perplexity from 15.41 to 14.40 over a parameter-matched Transformer. These gains extend across architectural variants. The Lattice composes with prior residual-stream enhancements: with hyper-connections and input residuals, it lowers validation perplexity from 15.41 (base Transformer) and 14.22 (Transformer…
Microsoft's chief executive warned that the rise of artificial general intelligence presents societal risks far beyond standard tech sector rivalries. If a single dominant model monopolizes critical industries, it could eliminate labor and threaten democratic institutions. To counteract this consolidation of power, enterprise environments must promote the widespread distribution of autonomous systems rather than rely on centralized control.
Developers attempting to access the official project codebase have encountered unexpected missing pages rather than the standard repository. The abrupt absence has left users without direct access to the source code hosted under the corporate account. Observers are awaiting further clarification on the reason behind the removal of the critical machine learning framework.
Interim leadership moved to reassure investors that recent executive upheaval will not disrupt the company's broader platform initiatives or major pending transactions. The chipmaker aims to maintain continuity by backing freely accessible architectures to fuel enterprise adoption across its hardware stack. Leadership stability remains under scrutiny, however, as federal authorities and lawmakers review the circumstances surrounding the sudden transition.