Attention-Aware Routing: Coupling Routing and Attention in MoEs
arXiv:2609.20974v1 Announce Type: new Abstract: In Mixture-of-Experts language models, the router typically selects and weights experts based on the token's hidden state, utilizing limited contextual information. We propose Attention-Aware...
Read more: https://arxiv.org/abs/2609.20974
Scan with your phone
Scan with your phone