ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

LLM Systems 18: Memory Optimization in Distributed Training