TeleChat-12B-FP16-Unofficial

Papers using or citing this model 4

• RoFormer: Enhanced Transformer with Rotary Position Embedding 6.8K citations · 2021 · arXiv (Cornell University) · arXiv

  Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various...

• GLU Variants Improve Transformer 2.4K citations · 2020 · arXiv (Cornell University) · arXiv

  Gated Linear Units (arXiv:1612.08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear)...

• Root Mean Square Layer Normalization 2.1K citations · 2019 · arXiv (Cornell University) · arXiv

  Layer normalization (LayerNorm) has been successfully applied to various deep neural networks to help stabilize training and boost model convergence because of its capability in handling re-centering and re-scaling of both inputs and...

• TeleChat Technical Report 14 citations · 2024 · arXiv

  In this technical report, we present TeleChat, a collection of large language models (LLMs) with parameters of 3 billion, 7 billion and 12 billion. It includes pretrained language models as well as fine-tuned chat models that is aligned...