Python script for manipulating the existing tokenizer.
The solution was tested on Llama3-8B tokenizer.
Installation from PyPI:
pip install tokenizerchanger- Python 3.9+
- tokenizers>=0.21.0
- transformers>=4.47.0
- tqdm>=4.66.4
Python script for manipulating the existing tokenizer.
The solution was tested on Llama3-8B tokenizer.
Installation from PyPI:
pip install tokenizerchanger