Artificial Intelligence1 min read

Attention to Mamba: A Recipe for Cross-Architecture Distillation

By · Published by Everything Blog

In short

The two-stage distillation approach transitions knowledge from a conventional Transformer into a linearized Attention model via a kernel trick before converting it into an adapted Mamba model. During validation with a Pythia-1B teacher, the compressed student model achieved a competitive perplexity of 14.01 compared to the original 13.86. Although the distilled model retained most capabilities on downstream evaluations, researchers observed noticeable performance drops during tasks requiring intensive in-context recall.

Key points

  • Knowledge transfer between fundamentally different model architectures is viable through…: Knowledge transfer between fundamentally different model architectures is viable through an intermediate linearized Attention phase.
  • Converting pretrained Transformers into pure Mamba architectures unlocks lower memory usa…: Converting pretrained Transformers into pure Mamba architectures unlocks lower memory usage and faster token generation.
  • Tasks that rely heavily on in-context recall remain a primary performance bottleneck for…: Tasks that rely heavily on in-context recall remain a primary performance bottleneck for distilled State Space Models.

Sources checked by our analysis pipeline: 1

AuthorsAbhinav Moudgil†**, Ningyuan Huang‡**, Eeshan Gunesh Dhekane, Pau Rodríguez Lopez, Luca Zappella, Federico Danieli

Attention to Mamba: A Recipe for Cross-Architecture Distillation

AuthorsAbhinav Moudgil†**, Ningyuan Huang‡**, Eeshan Gunesh Dhekane, Pau Rodríguez Lopez, Luca Zappella, Federico Danieli

State Space Models (SSMs) such as Mamba have become a popular alternative to Transformer models, due to their reduced memory consumption and higher throughput at generation compared to their Attention-based counterparts. On the other hand, the community has built up a considerable body of knowledge on how to train Transformers, and many pretrained Transformer models are readily available. To facilitate the adoption of SSMs while leveraging existing pretrained Transformers, we aim to identify an effective recipe to distill an Attention-based model into a Mamba-like architecture. In prior work on cross-architecture distillation, however, it has been shown that a naïve distillation procedure from Transformers to Mamba fails to preserve the original teacher performance, a limitation often overcome with hybrid solutions combining Attention and SSM blocks. The key argument from our work is that, by equipping Mamba with a principled initialization, we can recover an overall…

Artificial Intelligence

More in Artificial Intelligence