arcsin2's blog
  • 首页
  • 归档
  • 分类
  • 标签
  • 关于
  • 友链
  • English

LLM Systems 21: Parameter Efficient Fine-Tuning for LLM

0. Recap Direct quantization for low-bit numbers absmax: linearly scale according to max abs value zero-point: finding zero-point and scale Layer-wise quantization approaches AdaQuant / K
2026-11-14
ML System
#CMU 11868 #LLM System #LoRA

LLM Systems 20: LLM Quantization: GPTQ

0. Recap low-bit number representation in computer BF16: 16-bit half precision floating point numbers, better for ML int8 Direct quantization Absmax: linearly scale according to max abs
2026-11-14
ML System
#CMU 11868 #LLM System #LLM Quantizatio

LLM Systems 19: LLM Quantization: Basic methods

0. Recap 之前课程主线: LLM 简介 CUDA 编程基础 深度学习框架(自动求导) Transformer/Pre-trained LLM LLM 前后两个辅助组件:Tokenizer 与 Decoding Transformer 加速:Kernel Fusion & Mix-precision & Memory Management Optimiza
2026-11-14
ML System
#CMU 11868 #LLM System #LLM Quantizatio

LLM Systems Assignment 5: Distributed Training and Fine-Tuning

本次 homework 的任务是:实现 DP/PP/LoRA 的核心代码,整体代码量非常少,本文对实现要点和一些注意事项进行介绍。 1. Data Parallel DP 部分的待实现代码非常简单,唯一需要提一点的是 ranndom 的相关使用方法:作业规定了不能使用普通的 random.shuffle 而是必须手动从指定 seed 构造 gen = random.Random(seed
2026-10-31
ML System
#CMU 11868 #LLM System #Distributed Training

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models arXiv 提交日期:4 Oct 2019(Megatron-LM v1 paper 提交约 2 周后,即:17 Sep 2019 -> 4 Oct 2019) 代码:DeepSpeed ZeRO 作者: 摘要 大型深度
2026-10-31
ML System
#CMU 11868 #LLM System #Distributed Training #ZeRO

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models arXiv 提交日期:11 Jan 2024 GitHub:DeepSeek-MoE 作者: 摘要 在大语言模型时代,混合专家(MoE)架构是在扩展模型参数规模时控制计算成本的一项极具前景的
2026-10-31
ML System
#CMU 11868 #LLM System #Distributed Training #MoE

DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale

DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale arXiv 提交日期:14 Jan 2022(发表于 ICML 2022) 代码:DeepSpeed 作者: 摘要 随着当下超大密集模型(Dense Models)的
2026-10-31
ML System
#CMU 11868 #LLM System #Distributed Training #MoE

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity arXiv 发布日期:11 Jan 2021(发表于:Journal of Machine Learning Research 2022) 代码:t5x 作者: 摘要 在深度学习中,
2026-10-31
ML System
#CMU 11868 #LLM System #Distributed Training #MoE

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding arXiv 提交日期:30 Jun 2020(发表于 ICLR 2021) 作者: 摘要 神经网络规模扩展(Scaling) 在拥有海量训练数据和计算资源的许多现实机器学习应用中,对提升模型质量起到了至关重要的
2026-10-31
ML System
#CMU 11868 #LLM System #Distributed Training #MoE

Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM

Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM arXiv 提交日期:9 Apr 2021 代码:Megatron-LM 作者: 摘要: 大语言模型在多项任务中均取得了当前最佳(SOTA)的准确率。然而,高效地训练这些模型面临着严峻挑战,原因在于: a) G
2026-10-31
ML System
#CMU 11868 #LLM System #Distributed Training #3D Parallel
1234…11

搜索

Hexo Fluid
总访问量 次 总访客数 人
苏ICP备2021026429号-1