DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

LLM Systems 17: System for MoE Models