共计 14 篇文章
2026
LLM Systems Assignment 5: Distributed Training and Fine-Tuning
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism
LLM Systems 18: Memory Optimization in Distributed Training