starcoder2-15b

Papers using or citing this model 4

• Longformer: The Long-Document Transformer 6.1K citations · 2020 · arXiv (Cornell University) · arXiv

  Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention...

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness 5.6K citations · 2022 · arXiv (Cornell University) · arXiv

  Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length. Approximate attention methods have attempted to address this problem by trading off...

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints 1.9K citations · 2023 · arXiv (Cornell University) · arXiv

  Multi-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not be desirable to train a separate model just for faster...

• Efficient Training of Language Models to Fill in the Middle 340 citations · 2022 · arXiv (Cornell University) · arXiv

  We show that autoregressive language models can learn to infill text after we apply a straightforward transformation to the dataset, which simply moves a span of text from the middle of a document to its end. While this data...