From e9cdd7581e0c562e343604a173f237fa40e9dd33 Mon Sep 17 00:00:00 2001 From: Hk669 Date: Tue, 28 May 2024 02:28:17 +0530 Subject: [PATCH] update usage --- README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/README.md b/README.md index 376559b..eda5747 100644 --- a/README.md +++ b/README.md @@ -35,6 +35,13 @@ Every LLM(LLama, Gemini, Mistral..) use their own Tokenizers trained on their ow this tutorial leverages the `special_tokens` usage in the Tokenizer. +Install the package + +```shell +pip install bpetokenizer +``` + + ```py from bpetokenizer import BPETokenizer