The 36.2-million-parameter model was pretrained from scratch on an Apple Silicon MacBook Air with 24 GB of memory. Across 12,000 steps on public-domain detective fiction, the system processed nearly 98 million tokens at roughly 10.7k tokens per second. The training run reduced validation loss from 5.02 to 3.51 while confirming that auxiliary-loss-free balancing keeps expert utilization stable. Complete code, setup instructions, dataset preparation scripts, and trained checkpoints have been released publicly for the community.
Key points
Routing bias alone successfully balanced expert utilization between 5.7% and 6.9% without…: Routing bias alone successfully balanced expert utilization between 5.7% and 6.9% without degrading the model's validation loss.
The project demonstrates that complex state-of-the-art model architectures can be faithfu…: The project demonstrates that complex state-of-the-art model architectures can be faithfully reproduced on consumer-grade hardware.
The release is designed as an open-source educational demonstration rather than a product…: The release is designed as an open-source educational demonstration rather than a production-ready, high-capability language model.
Sources checked by our analysis pipeline: 1
The architecture from the DeepSeek-V3 technical report in 190 lines of pure JAX, pretrained from scratch on a MacBook Air on public-domain detective fiction.
No Flax, no Equinox, no PyTorch. Params are plain dicts of arrays and every part of the model is a function you can read top to bottom in deepseek_v3.py
.
Read the write-up: DeepSeek V3 in 190 lines of pure JAX, with diagrams, the code explained piece by piece, and the results.
Idea from the paper | What it does | Where |
---|---|---|
Multi-head Latent Attention (MLA) | Caches one small latent per token instead of full keys and values: 480 → 144 numbers per token per layer here | mla , init_cache |
Decoupled RoPE | One rotary key shared by all heads, kept outside the compressed latent | rope , apply_rope , mla |
DeepSeekMoE | 1 shared expert that always runs + 16 routed experts, top-4 per token | route , moe |
Auxiliary-loss-free balancing | A per-expert bias nudges routing toward idle experts; it changes which experts are picked, not their…
Soaring infrastructure expenses, heightened regulatory scrutiny, and severe execution delays are driving widespread attrition across the tech sector. Both emerging ventures and established giants are being forced to shutter stand-alone tools or consolidate features into existing ecosystems. This broad retrenchment highlights the growing difficulty of converting ambitious technological concepts into financially viable, consumer-ready products.
Executive turnover at Nvidia alongside a political shift on Capitol Hill is prompting a strategic realignment across the artificial intelligence sector. While the semiconductor giant faces its longest weekly losing streak in over a decade, rival chipmakers and major hyperscalers are securing significant market gains. Tighter anticipated oversight of new data-center construction is simultaneously increasing the premium on currently operating infrastructure amid persistent global hardware shortages.
Campaigners, environmental organisations, and the Scottish Greens are demanding a moratorium on large-scale digital infrastructure as public backlash spreads across Europe. Critics warn that massive facilities could overwhelm Scotland's renewable energy supply, inflate consumer utility bills, and compromise national climate targets. Meanwhile, Scottish officials maintain that strict planning oversight will prevent unchecked development while capturing the industry's economic potential.