LookAtMyQuant compressed the Inkling-2-Mini model from 18.89 GB to just 5.38 GB at 1.63 bits per weight while beating the larger 9.14 GB Unsloth release on key coding and math benchmarks. The compiler selectively compresses specific model segments to preserve critical capabilities, enabling an 8.87 GB variant to fully match the uncompressed model's 148 score on GSM8K. Using custom Metal kernels on an Apple M5 Max chip, the 5.4 GB model hits a median decode speed of 125.2 tokens per second alongside a 4-bit DSpark drafter. The same post-training framework has also scaled down other networks, including reducing Qwen3.8-27B to 7.2 GB.
Key points
Proximal fine-tuning paired with trellis codes allows extreme low-bit model quantization…: Proximal fine-tuning paired with trellis codes allows extreme low-bit model quantization without sacrificing foundational benchmark accuracy.
Custom runtime kernels enable rapid local inference on Apple silicon hardware while consu…: Custom runtime kernels enable rapid local inference on Apple silicon hardware while consuming under six gigabytes of active RAM.
LookAtMyQuant's post-training quantization techniques apply broadly to diverse architectu…: LookAtMyQuant's post-training quantization techniques apply broadly to diverse architectures, successfully shrinking Qwen models down to fractionary memory footprints.
Sources checked by our analysis pipeline: 1
LookAtMyQuant is a quantization compiler. It measures how much each stored piece of a model hurts the output when compressed. It then picks a code for each piece so the whole model fits a size budget with the least damage.
On Gemma 4 E2B, its GGUF has 12× lower KL than Q4_K_M at the same file size. The method and every result are on the site.
The code is under the MIT license.
LookAtMyQuant 2 takes Qwen3.5-4B to 1.23 GB, 2.3 bits per weight. The file solves 330 of 540 exact-answer math problems. Prism ML's Bonsai 2 Ternary 4B solves 328 at 1.20 GB, and the original solves 359. Its KL from the original on held-out text is 0.45 nats, against 0.59 for Bonsai.
The recipe is post-training quantization with trellis codes, calibrated on the model's own worked math solutions with digit tokens weighted 10×, and with bits allocated per row by Fisher information. 4,000 steps of distillation follow, in which every layer is re-quantized once from the original weights. The page has the findings, the full recipe and its…
Industry observers and analysts are actively debating which groups will capture the true value created by advanced automated systems. Uncertainty persists over how the technology's broader impacts will distribute advantages across various sectors and end users. Resolving this issue remains critical as adoption deepens worldwide.
Jensen Huang's fall represents bad news for accelerationists in Silicon Valley. Further details regarding the impact of these developments have not yet been provided.