David Nellans
4 papers 106 citations h-index 3 2019–2025
GPU Domain Specialization via Composable On-Package Architecture
As GPUs scale their low precision matrix math throughput to boost deep learning (DL) performance, they upset the balance between math throughput and memory system capabilities. We demonstrate that...
Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training
Deploying deep learning (DL) models across multiple compute devices to train large and complex models continues to grow in importance because of the demand for faster and more frequent training....
The Architectural Implications of Distributed Reinforcement Learning on CPU-GPU Systems
With deep reinforcement learning (RL) methods achieving results that exceed human capabilities in games, robotics, and simulated environments, continued scaling of RL training is crucial to its...
Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens
Expert Parallelism (EP) permits Mixture of Experts (MoE) models to scale beyond a single GPU. To address load imbalance across GPUs in EP, existing approaches aim to balance the number of tokens...