Hady W. Lauw

62 papers 6 citations h-index 1 2017–2026

A Partition Cover Approach to Tokenization

Tokenization is the process of encoding strings into tokens of a fixed vocabulary size, and is widely utilized in Natural Language Processing applications. The leading tokenization algorithm today...