Wenxiang Lin

FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models

Recent large language models (LLMs) have tended to leverage sparsity to reduce computations, employing the sparsely activated mixture-of-experts (MoE) technique. MoE introduces four modules,...

Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism

The mixture-of-experts (MoE) architecture scales model size with sublinear computational increase but suffers from memory-intensive inference due to KV caches and sparse expert activation. Recent...

Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules

Sparsely-activated Mixture-of-Expert (MoE) layers have found practical applications in enlarging the model size of large-scale foundation models, with only a sub-linear increase in computation...

Task Scheduling for Efficient Inference of Large Language Models on Single Moderate GPU Systems

Large language models~(LLMs) are known for their high demand on computing resources and memory due to their substantial model size, which leads to inefficient inference on moderate GPU systems....

Accelerating MoE Training through Expert Placement and Replica Coordination

Distributed training of sparsely activated mixture-of-experts (MoE) models is often bottlenecked by workload imbalance and all-to-all (A2A) communication induced by dynamic token routing. Existing...

ScheInfer: Efficient Inference of Large Language Models with Task Scheduling on Moderate GPUs

CoLOR-DP: Conjugate Low-Rank Differential Privacy for Structure-Aware LoRA Fine-Tuning

Mast: Efficient Training of Mixture-of-Experts Transformers with Task Pipelining and Ordering

Mitigating Contention in Stream Multiprocessors for Pipelined Mixture of Experts: An SM-Aware Scheduling Approach

SP-MoE: Expediting Mixture-of-Experts Training with Optimized Pipelining Planning

Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules

HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap

ECascade-RCNN: Enhanced Cascade RCNN for Multi-scale Object Detection in UAV Images

VisDrone-DET2020: The Vision Meets Drone Object Detection in Image Challenge Results

ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training

KernelFlume: Elastic Core-Attention Scaling for Agentic Long-Context Decoding

LLM serving is increasingly dominated by long and dynamic decode workloads from agents, reasoning models, and extended conversations. When bursty long-context demand exceeds deployed capacity,...

AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs

Quantization is a key method for reducing the GPU memory requirement of training large language models (LLMs). Yet, current approaches are ineffective for 4-bit activations and 8-bit gradients,...

ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training

Communication has emerged as a critical bottleneck in the distributed training of large language models (LLMs). While numerous approaches have been proposed to reduce communication overhead, the...