The model provides base weights captured immediately following pre-training, entirely omitting mid- or post-training adjustments. It incorporates an architecture featuring 43 Transformer layers, a hidden size of 4096, and 64 attention heads. To keep hardware requirements viable during large-context operations, the model balances Multi-head Latent Attention with Heavily Compressed Attention layers. This structural setup facilitates extensive fine-tuning across long context windows without triggering prohibitive resource demands.
Key points
Researchers can pursue specialized model customization without having to fund costly init…: Researchers can pursue specialized model customization without having to fund costly initial pre-training runs.
Heavily Compressed Attention layers reduce KV-cache footprints by compressing every 128 t…: Heavily Compressed Attention layers reduce KV-cache footprints by compressing every 128 tokens into one entry.
The checkpoint strictly provides pre-trained base weights that have not undergone post-tr…: The checkpoint strictly provides pre-trained base weights that have not undergone post-training alignment or instruction tuning.
Sources checked by our analysis pipeline: 1
Table of Contents
1. Introduction
A base model is valuable both for the capabilities that pre-training builds and for the starting point it offers for further training. Different domains, reasoning tasks and agentic applications call for different data, training objectives and behaviors. Releasing the weights at the end of pre-training lets developers build on the general capabilities already in place and choose for themselves where to take the model next.
Today we are releasing ERNIE-5.5-Flash-Base, a sparse Mixture-of-Experts (MoE) language model with 280B total parameters, 14B of which are activated per token. It was pre-trained from scratch on 33.5T tokens with a training context of up to 128K, and supports a 1M-token context for fine-tuning and inference. The weights are the checkpoint at the end of pre-training, before any mid- or post-training, and can serve as a common starting point for domain adaptation, instruction tuning and reinforcement-learning research.
To be worth building on, a base model has to stay affordable to train and run, so efficiency was a design goal from the outset. Further training samples from the model repeatedly,…
The commitments follow widespread pushback from voters who rejected local artificial intelligence infrastructure measures across the country due to fears of rising utility bills, job losses, and environmental damage. The artificial intelligence developer aims to restore public trust by increasing transparency and urging lawmakers to codify these operational standards into federal law.
Singapore's primary publisher is deploying targeted artificial intelligence models to bridge linguistic divides across the nation's four official languages. By converting premier journalism between English and Chinese, the organization aims to broaden community access to local reporting without sacrificing distinct brand voices. The strategy combines automated translation technology with human oversight to ensure cultural accuracy for diverse audiences.
Public resistance to technology infrastructure surged on Tuesday as voters in multiple states struck down ballot measures for artificial intelligence data centers. This widespread pushback reflects broader American skepticism that casts the technology as harmful and politically entangled. The anti-tech sentiment poses new uncertainties for health care policymakers preparing to navigate critical regulatory debates in the upcoming legislative session.