Hugging Peers

tokenization

☆ Save
by Ceceliachenen word-segmentation Pytorch Apache License 2.0
2.2K downloads 0 likes on ModelScope updated

ModelScope

Ceceliachenen/tokenization · model repository official

Open upstream ↗
Revision
master
Files
4
Downloaded from this source
2.2K

Provider-specific. This path, revision and file list belong to ModelScope alone. A different source for the same model may publish different files, or the same names with different contents — which is why nothing here is merged across providers.

Files 4 · revision master

Hashes: Git OID identifies the file in the repository; LFS SHA256 is the fingerprint of a large file's contents; Xet is the provider's own content fingerprint. Expand “hashes” on any row for the full values.

  • .gitattributes 1.6 KiB abb4b1aa6545
    hashes
    LFS SHA256
    abb4b1aa654526de5abde955a3c42787604c95baee0656216cd4498fcd12aecf
  • configuration.json 50 B d9038adc5659
    hashes
    LFS SHA256
    d9038adc565992beb36e909f8f352664f1e0126feb50cf10dbb2ba10de083ff0
  • qwen.tiktoken 2.4 MiB b2b1b8dfb5cc
    hashes
    LFS SHA256
    b2b1b8dfb5cc5f024bafc373121c6aba3f66f9a5a0269e243470a1de16a33186
  • README.md 1.4 KiB 74984e00bdfd
    hashes
    LFS SHA256
    74984e00bdfdfd551ba3b1dea65e14f3f1aa01fadefa4481c46f509005d30dab