dbrx-base-converted-v2

Papers using or citing this model 2

• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel 936 citations · 2023 · arXiv

  It is widely acknowledged that large models have the potential to deliver superior performance across a broad range of domains. Despite the remarkable progress made in the field of machine learning systems research, which has enabled...

• MegaBlocks: Efficient Sparse Training with Mixture-of-Experts 291 citations · 2022 · arXiv (Cornell University) · arXiv

  We present MegaBlocks, a system for efficient Mixture-of-Experts (MoE) training on GPUs. Our system is motivated by the limitations of current frameworks, which restrict the dynamic routing in MoE layers to satisfy the constraints of...