Improve Tokenizer Class: Error Handling, Flexibility

[zeelsheladiya]

Input Validation and Error Handling:

• Added input validation checks for the bos and eos parameters in the encode method to ensure they are boolean values.
• Enhanced error messages for better context and debugging.

Flexible Model Loading:

• Modified the constructor of the Tokenizer class to optionally accept a model path or URL.
• Users can now load models from URLs or Hugging Face model identifiers, making it more versatile for different deployment scenarios.

Handling Unknown Tokens:

• Improved tokenization by handling unknown tokens using SentencePiece's unk_id. Tokens outside the vocabulary range are replaced with the <UNK> token.

These changes contribute to the overall reliability and usability of the Tokenizer class, enabling smoother integration into various projects.

[facebook-github-bot]

Hi @zeelsheladiya!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

[zeelsheladiya]

Hi @DamienAllonsius ,

I hope you're doing well! I wanted to let you know that I've made the requested change to the code. I replaced the usage of Union[str, None] with Optional[str] as per your feedback.

You can review the updated code at. I'd greatly appreciate it if you could take a look and let me know if everything looks good to you now. If you have any further suggestions or feedback, please feel free to share them.

Thank you for your time and guidance throughout this process. I look forward to hearing your thoughts on the changes.

[DamienAllonsius]

Hi @zeelsheladiya I solved the conflicts with main.
Any idea of how we could make Sentenpiece.encode return an unk_id character?

[zeelsheladiya]

Hi @DamienAllonsius,

As per my Understanding, The SentencePieceProcessor.encode method returns an out-of-vocabulary token as a separate token, rather than mapping it to the unk_id character. If we want to make SentencePiece.encode return the unk_id character when it encounters an unknown token, we can modify the encode method.

We can use a list comprehension to check each token ID returned by self.sp_model.encode(s). If the token ID is within the valid range of token IDs (i.e., it's a known token), we keep it as is. If the token ID is not within the valid range, we replace it with self.unk_id, effectively mapping unknown tokens to the unk_id character.

Please Let me know your thoughts. :)

[DamienAllonsius]

LGTM, tested the code but could not generate any character with unk_id encoding

[ruanslv]

Why make this optional? How would you initialize the tokenizer without the model file?

[ruanslv]

Would you have an example of an input string that requires this?

[ruanslv]

I don't think we need this, the exception itself should be clear enough?