ReluLLaMA-13B

Papers using or citing this model 2

• MoEfication: Transformer Feed-forward Layers are Mixtures of Experts 215 citations · 2021 · Findings · arXiv

  Recent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. However, the computational patterns of FFNs are still unclear. In this work, we...

• ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models 138 citations · 2023 · arXiv

  Large Language Models (LLMs) with billions of parameters have drastically transformed AI applications. However, their demanding computation during inference has raised significant challenges for deployment on resource-constrained...