Skip to content

Commit

Permalink
update usage
Browse files Browse the repository at this point in the history
  • Loading branch information
Hk669 committed May 27, 2024
1 parent 707d65b commit e9cdd75
Showing 1 changed file with 7 additions and 0 deletions.
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,13 @@ Every LLM(LLama, Gemini, Mistral..) use their own Tokenizers trained on their ow

this tutorial leverages the `special_tokens` usage in the Tokenizer.

Install the package

```shell
pip install bpetokenizer
```


```py
from bpetokenizer import BPETokenizer

Expand Down

0 comments on commit e9cdd75

Please sign in to comment.