Artificial Intelligence1 min read

Inkling-2-Mini in 5.4 GB Memory

By · Published by Everything Blog

In short

LookAtMyQuant compressed the Inkling-2-Mini model from 18.89 GB to just 5.38 GB at 1.63 bits per weight while beating the larger 9.14 GB Unsloth release on key coding and math benchmarks. The compiler selectively compresses specific model segments to preserve critical capabilities, enabling an 8.87 GB variant to fully match the uncompressed model's 148 score on GSM8K. Using custom Metal kernels on an Apple M5 Max chip, the 5.4 GB model hits a median decode speed of 125.2 tokens per second alongside a 4-bit DSpark drafter. The same post-training framework has also scaled down other networks, including reducing Qwen3.8-27B to 7.2 GB.

Key points

  • Proximal fine-tuning paired with trellis codes allows extreme low-bit model quantization…: Proximal fine-tuning paired with trellis codes allows extreme low-bit model quantization without sacrificing foundational benchmark accuracy.
  • Custom runtime kernels enable rapid local inference on Apple silicon hardware while consu…: Custom runtime kernels enable rapid local inference on Apple silicon hardware while consuming under six gigabytes of active RAM.
  • LookAtMyQuant's post-training quantization techniques apply broadly to diverse architectu…: LookAtMyQuant's post-training quantization techniques apply broadly to diverse architectures, successfully shrinking Qwen models down to fractionary memory footprints.

Sources checked by our analysis pipeline: 1

LookAtMyQuant is a quantization compiler. It measures how much each stored piece of a model hurts the output when compressed. It then picks a code for each piece so the whole model fits a size budget with the least damage.

On Gemma 4 E2B, its GGUF has 12× lower KL than Q4_K_M at the same file size. The method and every result are on the site.

The code is under the MIT license.

LookAtMyQuant 2 takes Qwen3.5-4B to 1.23 GB, 2.3 bits per weight. The file solves 330 of 540 exact-answer math problems. Prism ML's Bonsai 2 Ternary 4B solves 328 at 1.20 GB, and the original solves 359. Its KL from the original on held-out text is 0.45 nats, against 0.59 for Bonsai.

The recipe is post-training quantization with trellis codes, calibrated on the model's own worked math solutions with digit tokens weighted 10×, and with bits allocated per row by Fisher information. 4,000 steps of distillation follow, in which every layer is re-quantized once from the original weights. The page has the findings, the full recipe and its…

Artificial Intelligence