NVIDIA 谈生物基础模型的高效 MoE 训练

Efficient MoE Training for Biological Foundation Models

AI 导读

NVIDIA 介绍面向生物基础模型的高效 MoE 训练方法，指出稠密 Transformer 中每个 token 都要过每一层，扩容会同时推高训练与推理算力；MoE 则用多个子网络（专家）替代，每个 token 只激活其中一小部分。

正文

DNA. As language models grow, scaling dense architectures becomes increasingly expensive. In a dense transformer, every token passes through every layer, so adding... DNA.

As language models grow, scaling dense architectures becomes increasingly expensive. In a dense transformer, every token passes through every layer, so adding capabilities increases computation for both training and inference. Mixture-of-experts (MoE) architectures take a different approach to scaling by using many subnetworks, or experts, while activating only a small subset for each token.

Source

来源：NVIDIA Developer Blog · developer.nvidia.com