Error when converting llama3.2-1B tokenizer.model to tokenizer.bin

[a8nova]

🐛 Describe the bug

When following instructions from https://github.com/pytorch/executorch/blob/main/examples/demo-apps/apple_ios/LLaMA/docs/delegates/xnnpack_README.md and when trying to convert llama3.2-1B tokenizer to bin you will see below error:

(et_xnnpack) /content/HI/executorch# python -m extension.llm.tokenizer.tokenizer -t /root/.llama/checkpoints/Llama3.2-1B/tokenizer.model -o tokenizer.bin
Traceback (most recent call last):
  File "/usr/local/envs/et_xnnpack/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/envs/et_xnnpack/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/content/HI/executorch/extension/llm/tokenizer/tokenizer.py", line 140, in <module>
    t = Tokenizer(args.tokenizer_model)
  File "/content/HI/executorch/extension/llm/tokenizer/tokenizer.py", line 26, in __init__
    self.sp_model = SentencePieceProcessor(model_file=model_path)
  File "/usr/local/envs/et_xnnpack/lib/python3.10/site-packages/sentencepiece/__init__.py", line 468, in Init
    self.Load(model_file=model_file, model_proto=model_proto)
  File "/usr/local/envs/et_xnnpack/lib/python3.10/site-packages/sentencepiece/__init__.py", line 961, in Load
    return self.LoadFromFile(model_file)
  File "/usr/local/envs/et_xnnpack/lib/python3.10/site-packages/sentencepiece/__init__.py", line 316, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
RuntimeError: Internal: could not parse ModelProto from /root/.llama/checkpoints/Llama3.2-1B/tokenizer.model

[HSANGLEE]

Dear @a8nova

As far as I know, from LLaMA3 you can just use tokenizer.model file not tokenizer.bin.

[a8nova]

Thank you, @HSANGLEE. I am currently blocked by #5840