1.Introduction
Current large language models (LLMs) primarily utilize next-token prediction method for inference, which significantly impedes their processing speed. In this paper, we introduce a novel inference methodology termed next-sentence prediction, aiming at enhancing the inference efficiency of LLMs. We present Sentence Variational Autoencoder (SentenceVAE), which includes a Sentence Encoder to compress multiple tokens in a sentence into a single token, and a Sentence Decoder to reconstruct it.
Fig. 1. The schematic form of SentenceVAE.
By integrating SentenceVAE into the input and output layers of LLMs, we develop Sentence-level LLMs (SLLMs) that employ a sentence-by-sentence inference method.
Fig. 2. (a) The schematic form of published LLMs. (b) The schematic form of SLLMs, which embedded with SentenceVAEs.
The SLLMs can maintain the integrity of the original semantic content by segmenting the context into sentences, thereby improving accuracy while boosting inference speed. Moreover, compared to previous LLMs, SLLMs process fewer tokens over equivalent context length, significantly reducing memory demands for self-attention computation and facilitating the handling of longer context. Extensive experiments on Wanjuan dataset have revealed that the proposed method can accelerate inference speed by 204 ~ 365%, reduce perplexity (PPL) to 46 ~ 75% of its original metric, and decrease memory overhead by 86 ~ 91% for the equivalent context length, compared to previous token-by-token methods.
| Model | Total Params | Average PPL | Mean output throughput (toks/s) | Mean GPU memory (KB/token) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| OPT↓ | SLLM↓ | Δ↓ | OPT↑ | SLLM↑ | Δ↑ | OPT↓ | SLLM↓ | Δ↓ | ||
| SLLM-125M-H1 | 214M | 26.75 | 31.68 | +18.4% | 214.57 | 652.78 | +204.2% | 73.15 | 12.03 | -83.6% |
| SLLM-125M-H2 | 226M | 44.60 | +66.7% | 539.80 | +151.6% | 7.08 | -90.3% | |||
| SLLM-125M-H4 | 250M | 14.32 | -46.5% | 332.12 | +54.8% | 10.00 | -86.3% | |||
| SLLM-350M-H1 | 429M | 25.18 | 24.84 | -1.4% | 144.33 | 481.39 | +233.5% | 197.59 | 29.98 | -84.8% |
| SLLM-350M-H2 | 450M | 14.81 | -41.2% | 442.23 | +206.4% | 26.78 | -86.4% | |||
| SLLM-350M-H4 | 492M | 10.17 | -59.6% | 315.61 | +118.7% | 17.73 | -91.0% | |||
| SLLM-1.3B-H1 | 1.61B | 15.95 | 8.76 | -45.1% | 119.07 | 479.71 | +302.9% | 400.01 | 57.07 | -85.7% |
| SLLM-1.3B-H2 | 1.69B | 3.84 | -75.9% | 553.95 | +365.2% | 55.14 | -86.2% | |||
In addition, by corroborating the Scaling Law, we extrapolated the feasibility of our methodologies to larger-scale models.
Fig. 3. Scaling Law of (a) SLLMs and (b) SVAEs.
2.Cite SentenceVAE
If you use SentenceVAE in your research, please cite our work by using the following BibTeX entry:
@article{an2024sentencevae,
title={SentenceVAE: Enable Next-sentence Prediction for Large Language Models with Faster Speed, Higher Accuracy and Longer Context},
author={An, Hongjun and Chen, Yifan and Sun, Zhe and Li, Xuelong},
journal={arXiv preprint arXiv:2408.00655},
year={2024}
}