deberta

Papers using or citing this model 2

• DeBERTa: Decoding-enhanced BERT with Disentangled Attention 4.2K citations · 2020 · arXiv (Cornell University) · arXiv

  Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with...

• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing 2.3K citations · 2021 · arXiv (Cornell University) · arXiv

  This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-training task. Our...