Python script for manipulating the existing tokenizer.
The solution was tested on Llama3-8B tokenizer.
Installation from PyPI:
pip install tokenizerchanger- Python 3.9+
- tokenizers>=0.21.0
- transformers>=4.47.0
- tqdm>=4.66.4
-
Notifications
You must be signed in to change notification settings - Fork 0
Python script for manipulating the existing tokenizer.
License
1kkiRen/Tokenizer-Changer
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
Python script for manipulating the existing tokenizer.