Update tokenizers dependency version to >=0.14#55

Merged

generall merged 2 commits intomainfrom tokenizers-upgrade

Nov 1, 2023

+195-50

Commits on Nov 1, 2023

* chore(pyproject.toml): update tokenizers dependency version to ^0.14.1
NirantK
committed
* chore(pyproject.toml): update tokenizers dependency version to be at least 0.14
NirantK
committed