CS 59200-MLS | Schedule

Date	Lecture	Readings	Presenters
Week#1 08/20	Lecture #1 Course Introduction [ slides ]		Xupeng Miao
Week#1 08/22	Lecture #2 Introduction to Deep Neural Networks [ slides ]		Xupeng Miao
Week#2 08/27	Lecture #3 Introduction to Deep Learning Systems [ slides ]		Xupeng Miao
Week#2 08/29	Lecture #4 Systems for Deep Learning [ slides ]	TensorFlow: A System for Large-Scale Machine Learning PyTorch: An Imperative Style, High-Performance Deep Learning Library (Optional) PyTorch Distributed: Experiences on Accelerating Data Parallel Training	Xupeng Miao
Week#3 09/03	Lecture #5 Systems for Classical ML [ slides ]	Scaling Distributed Machine Learning with the Parameter Server XGBoost: A Scalable Tree Boosting System	Sean Lee Xinji Jiang
Week#3 09/05	Lecture #6 ML Compilation [ slides ]	TVM: An Automated End-to-End Optimizing Compiler for Deep Learning MLIR: A Compiler Infrastructure for the End of Moore's Law (Optional) Halide: Decoupling Algorithms from Schedules for Easy Optimization of Image Processing Pipelines	Yanran Wu Raj Venkat Reddy Mavuram
Week#4 09/10	Lecture #7 ML Compilation [ slides ]	Learning to Optimize Tensor Programs Ansor: Generating High-Performance Tensor Programs for Deep Learning	Xinji Jiang Sarthak Mangla
Week#4 09/12	Lecture #8 Tensor Program Optimizations [ slides ]	TASO: Optimizing Deep Learning Computation with Automatic Generation of Graph Substitutions EINNET: Optimizing Tensor Programs with Derivation-Based Transformations (Optional) PET: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections	Shau-Shiun Chu Yunxin Sun
Week#5 09/17	Lecture #9 Data & Model Parallelism [ slides ]	Beyond Data and Model Parallelism for Deep Neural Networks Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism	Xupeng Miao
Week#5 09/19	Lecture #10 Pipeline Parallelism [ slides ]	GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism PipeDream: Generalized Pipeline Parallelism for DNN Training (Optional) Memory-Efficient Pipeline-Parallel DNN Training	Xupeng Miao
Week#6 09/24	Lecture #11 Device Placement Optimizations [ slides ]	Device Placement Optimization with Reinforcement Learning Efficient Algorithms for Device Placement of DNN Graph Operators	~~Shilpa Noushad~~ Saarang Srinivasan
Week#6 09/26	Lecture #12 Automatic Parallelization [ slides ]	Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism (Optional) Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization	Xupeng Miao
Week#7 10/01	Lecture #13 Memory Management: Rematerialization [ slides ]	Training Deep Nets with Sublinear Memory Cost Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization	Haoyu Jin Yanran Wu
Week#7 10/03	Lecture #14 Memory Management: ZeRO [ slides ]	ZeRO: Memory Optimizations Toward Training Trillion Parameter Models ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning (Optional) ZeRO-Offload: Democratizing Billion-Scale Model Training	Shige Liu WenYuan Huang
Week#8 10/10	Lecture #15 Recommendation Models [ slides ] Final Team Signup Due	HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework HET-GMP: A Graph-based System Approach to Scaling Large Embedding Model Training	Xupeng Miao
Week#9 10/15	Lecture #16 Graph Neural Networks [ slides ]	Reducing Communication in Graph Neural Network Training GNNAdvisor: An Adaptive and Efficient Runtime System for GNN Acceleration on GPUs	Kshitij Alwadhi ~~Yunxin Sun~~
Week#9 10/17	Lecture #17 Transformers and Language Models [ slides ]	Transformer: Attention is All you Need Training language models to follow instructions with human feedback	Xupeng Miao
Week#10 10/22	Lecture #18 Mixture-of-Experts Models [ slides ] One-page Proposal Due	FasterMoE: Modeling and Optimizing Training of Large-Scale Dynamic Pre-Trained Models FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement (Optional) GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding	Xupeng Miao
Week#10 10/24	Lecture #19 Heterogeneity-aware ML [ slides ]	More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce	Xupeng Miao
Week#11 10/29	Lecture #20 Systems for LLM Inference [ slides ] Intermediate Check-in Starting	Efficient Memory Management for Large Language Model Serving with PagedAttention SpecInfer: Accelerating Generative Large Language Model Serving with Speculative Inference and Token Tree Verification (Optional) Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems	Xupeng Miao
Week#11 10/31	Lecture #21 LLM Inference Requests Scheduling [ slides ]	Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving	Dwijen Chawra WenYuan Huang
Week#12 11/05	Lecture #22 Affordable ML Systems [ slides ]	SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient SpotServe: Serving Generative Large Language Models on Preemptible Instances	Xupeng Miao
Week#12 11/07	Lecture #23 Diffusion Models [ slides ]	DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models	~~Yunxin Sun~~ ~~Yunxin Sun~~
Week#13 11/12	Lecture #24 Model Compression [ slides ]	Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration	Sarthak Mangla ~~Shilpa Noushad~~
Week#13 11/14	Lecture #25 Data Management for ML [ slides ]	Distributed Deep Learning on Data Systems: A Comparative Analysis of Approaches Optimizing LLM Queries in Relational Workloads	~~Shige Liu~~ Haoyu Jin
Week#14 11/19	Lecture #26 ML Hardware [ slides ]	In-Datacenter Performance Analysis of a Tensor Processing Unit Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor	Shau-Shiun Chu Dwijen Chawra
Week#14 11/21	Lecture #27 ML for Systems [ slides ]	The Case for Learned Index Structures MemGPT: Towards LLMs as Operating Systems	Xupeng Miao Saarang Srinivasan
Week#15 11/26	Lecture #28 Federated Learning [ slides ]	Towards Federated Learning at Scale: System Design Oort: Efficient Federated Learning via Guided Participant Selection	Raj Venkat Reddy Mavuram Kshitij Alwadhi
Week#16 12/03	Lecture #29 Final Presentations [ slides ]		Group A Group B
Week#16 12/05	Lecture #30 Final Presentations [ slides ]		Group C Group D
12/12	Final Course Project Report Due