[BUG: Tokenizer Decode error during function calling]

[Mohrizk]

Python -VV

Python 3.11.9 | packaged by conda-forge | (main, Apr 19 2024, 18:36:13) [GCC 12.3.0]

Pip Freeze

absl-py @ file:///home/conda/feedstock_root/build_artifacts/absl-py_1705494584803/work
aiohttp @ file:///home/conda/feedstock_root/build_artifacts/aiohttp_1713964843148/work
aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1667935791922/work
annotated-types @ file:///home/conda/feedstock_root/build_artifacts/annotated-types_1716290248287/work
anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1717693030552/work
argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1692818318753/work
argon2-cffi-bindings @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi-bindings_1695386553988/work
arrow @ file:///home/conda/feedstock_root/build_artifacts/arrow_1696128962909/work
asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1698341106958/work
astunparse @ file:///home/conda/feedstock_root/build_artifacts/astunparse_1610696312422/work
async-lru @ file:///home/conda/feedstock_root/build_artifacts/async-lru_1690563019058/work
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1704011227531/work
Babel @ file:///home/conda/feedstock_root/build_artifacts/babel_1702422572539/work
beautifulsoup4 @ file:///home/conda/feedstock_root/build_artifacts/beautifulsoup4_1705564648255/work
bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1696630167146/work
blinker @ file:///home/conda/feedstock_root/build_artifacts/blinker_1715091184126/work
blis @ file:///home/conda/feedstock_root/build_artifacts/cython-blis_1696148823813/work
Bottleneck @ file:///work/ci_py311/bottleneck_1676827011612/work
Brotli @ file:///home/conda/feedstock_root/build_artifacts/brotli-split_1695989787169/work
cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work
cachetools @ file:///home/conda/feedstock_root/build_artifacts/cachetools_1708987703938/work
catalogue @ file:///home/conda/feedstock_root/build_artifacts/catalogue_1695626386490/work
certifi @ file:///home/conda/feedstock_root/build_artifacts/certifi_1718025014955/work/certifi
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1696001724357/work
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1698833585322/work
click @ file:///home/conda/feedstock_root/build_artifacts/click_1692311806742/work
cloudpathlib @ file:///home/conda/feedstock_root/build_artifacts/cloudpathlib-meta_1708995864616/work
colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1666700638685/work
comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1710320294760/work
confection @ file:///home/conda/feedstock_root/build_artifacts/confection_1701179076994/work
contourpy @ file:///home/conda/feedstock_root/build_artifacts/contourpy_1712429897138/work
cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography-split_1717559408348/work
cycler @ file:///home/conda/feedstock_root/build_artifacts/cycler_1696677705766/work
cymem @ file:///home/conda/feedstock_root/build_artifacts/cymem_1695443490958/work
debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1719378645730/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work
docstring_parser==0.16
entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1643888246732/work
exceptiongroup @ file:///home/conda/feedstock_root/build_artifacts/exceptiongroup_1704921103267/work
executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1698579936712/work
fastai @ file:///home/jhoward/miniconda3/conda-bld/fastai_1714256075359/work
fastcore @ file:///Users/jhoward/miniconda3/conda-bld/fastcore_1719125228499/work
fastdownload @ file:///home/jhoward/mambaforge/conda-bld/fastdownload_1657219113869/work
fastjsonschema @ file:///home/conda/feedstock_root/build_artifacts/python-fastjsonschema_1718477020893/work/dist
fastprogress @ file:///home/jhoward/mambaforge/conda-bld/fastprogress_1658473398631/work
filelock @ file:///home/conda/feedstock_root/build_artifacts/filelock_1719088281970/work
fire==0.6.0
flatbuffers @ file:///home/conda/feedstock_root/build_artifacts/python-flatbuffers_1711466727397/work
fonttools @ file:///home/conda/feedstock_root/build_artifacts/fonttools_1720359030008/work
fqdn @ file:///home/conda/feedstock_root/build_artifacts/fqdn_1638810296540/work/dist
frozenlist @ file:///home/conda/feedstock_root/build_artifacts/frozenlist_1702645450877/work
fsspec==2024.6.1
gast @ file:///home/conda/feedstock_root/build_artifacts/gast_1719403123000/work
gmpy2 @ file:///home/conda/feedstock_root/build_artifacts/gmpy2_1715527288181/work
google-auth @ file:///home/conda/feedstock_root/build_artifacts/google-auth_1719906113316/work
google-auth-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/google-auth-oauthlib_1688235217226/work
google-pasta==0.2.0
greenlet==3.0.3
grpcio @ file:///home/conda/feedstock_root/build_artifacts/grpc-split_1690942284331/work
h11 @ file:///home/conda/feedstock_root/build_artifacts/h11_1664132893548/work
h2 @ file:///home/conda/feedstock_root/build_artifacts/h2_1634280454336/work
h5py @ file:///home/conda/feedstock_root/build_artifacts/h5py_1717664837666/work
hpack==4.0.0
httpcore @ file:///home/conda/feedstock_root/build_artifacts/httpcore_1711596990900/work
httpx @ file:///home/conda/feedstock_root/build_artifacts/httpx_1708530890843/work
huggingface-hub==0.24.6
hyperframe @ file:///home/conda/feedstock_root/build_artifacts/hyperframe_1619110129307/work
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1713279365350/work
import-ipynb==0.1.4
importlib_metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1719361860083/work
importlib_resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1711040877059/work
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1719845459717/work
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1719582526268/work
ipywidgets @ file:///home/conda/feedstock_root/build_artifacts/ipywidgets_1716897651763/work
isoduration @ file:///home/conda/feedstock_root/build_artifacts/isoduration_1638811571363/work/dist
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1696326070614/work
Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1715127149914/work
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1714665484399/work
json5 @ file:///home/conda/feedstock_root/build_artifacts/json5_1712986206667/work
jsonpatch==1.33
jsonpointer @ file:///home/conda/feedstock_root/build_artifacts/jsonpointer_1718283368205/work
jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema-meta_1714573116818/work
jsonschema-specifications @ file:///tmp/tmpkv1z7p57/src
jupyter @ file:///home/conda/feedstock_root/build_artifacts/jupyter_1696255489086/work
jupyter-console @ file:///home/conda/feedstock_root/build_artifacts/jupyter_console_1678118109161/work
jupyter-events @ file:///home/conda/feedstock_root/build_artifacts/jupyter_events_1710805637316/work
jupyter-lsp @ file:///home/conda/feedstock_root/build_artifacts/jupyter-lsp-meta_1712707420468/work/jupyter-lsp
jupyter_client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1716472197302/work
jupyter_core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1710257359434/work
jupyter_server @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_1717122053158/work
jupyter_server_terminals @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_terminals_1710262634903/work
jupyterlab @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_1719418677442/work
jupyterlab_pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1707149102966/work
jupyterlab_server @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_server-split_1716433953404/work
jupyterlab_widgets @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_widgets_1716891641122/work
keras @ file:///home/conda/feedstock_root/build_artifacts/keras_1698427100715/work/keras-2.14.0-py3-none-any.whl#sha256=d7429d1d2131cc7eb1f2ea2ec330227c7d9d38dab3dfdf2e78defee4ecc43fcd
kiwisolver @ file:///home/conda/feedstock_root/build_artifacts/kiwisolver_1695379920604/work
langchain==0.2.16
langchain-core==0.2.38
langchain-text-splitters==0.2.4
langcodes @ file:///home/conda/feedstock_root/build_artifacts/langcodes_1714235526219/work
langsmith==0.1.114
language_data @ file:///home/conda/feedstock_root/build_artifacts/language-data_1714193818885/work
marisa-trie @ file:///home/conda/feedstock_root/build_artifacts/marisa-trie_1706566451123/work
Markdown @ file:///home/conda/feedstock_root/build_artifacts/markdown_1710435156458/work
markdown-it-py @ file:///home/conda/feedstock_root/build_artifacts/markdown-it-py_1686175045316/work
MarkupSafe @ file:///home/conda/feedstock_root/build_artifacts/markupsafe_1706899926732/work
matplotlib @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-suite_1715976244352/work
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1713250518406/work
mdurl @ file:///home/conda/feedstock_root/build_artifacts/mdurl_1704317613764/work
mistral_common==1.3.4
mistral_inference==1.3.1
mistune @ file:///home/conda/feedstock_root/build_artifacts/mistune_1698947099619/work
ml-dtypes @ file:///home/conda/feedstock_root/build_artifacts/ml_dtypes_1695280940315/work
mpmath @ file:///home/conda/feedstock_root/build_artifacts/mpmath_1678228039184/work
multidict @ file:///home/conda/feedstock_root/build_artifacts/multidict_1707040702345/work
munkres==1.1.4
murmurhash @ file:///home/conda/feedstock_root/build_artifacts/murmurhash_1695449783148/work
nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1710317608672/work
nbconvert @ file:///home/conda/feedstock_root/build_artifacts/nbconvert-meta_1718135430380/work
nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1712238998817/work
nbimporter==0.3.4
nest_asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1705850609492/work
networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1712540363324/work
notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1717767745914/work
notebook_shim @ file:///home/conda/feedstock_root/build_artifacts/notebook-shim_1707957777232/work
numexpr @ file:///home/conda/feedstock_root/build_artifacts/numexpr_1716812153765/work
numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1707225376651/work/dist/numpy-1.26.4-cp311-cp311-linux_x86_64.whl#sha256=d08e1c9e5833ae7780563812aa73e2497db1ee3bd5510d3becb8aa18aa2d0c7c
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.6.68
nvidia-nvtx-cu12==12.1.105
oauthlib @ file:///home/conda/feedstock_root/build_artifacts/oauthlib_1666056362788/work
opt-einsum @ file:///home/conda/feedstock_root/build_artifacts/opt_einsum_1696448916724/work
orjson==3.10.7
overrides @ file:///home/conda/feedstock_root/build_artifacts/overrides_1706394519472/work
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1718189413536/work
pandas @ file:///croot/pandas_1697477158611/work/dist/pandas-2.1.1-cp311-cp311-linux_x86_64.whl#sha256=ed5518777956b76c6290791e98ec9ff6ea64a621c757052a7697aa4252bc0d9b
pandocfilters @ file:///home/conda/feedstock_root/build_artifacts/pandocfilters_1631603243851/work
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1712320355065/work
pathlib==1.0.1
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1706113125309/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
pillow @ file:///home/conda/feedstock_root/build_artifacts/pillow_1712154447422/work
pkgutil_resolve_name @ file:///home/conda/feedstock_root/build_artifacts/pkgutil-resolve-name_1694617248815/work
platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1715777629804/work
ply @ file:///home/conda/feedstock_root/build_artifacts/ply_1712242996588/work
preshed @ file:///home/conda/feedstock_root/build_artifacts/preshed_1695644751322/work
prometheus_client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1707932675456/work
prompt_toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1718047967974/work
protobuf==4.21.12
psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1719274586160/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work
pyasn1 @ file:///home/conda/feedstock_root/build_artifacts/pyasn1_1713209357222/work
pyasn1_modules @ file:///home/conda/feedstock_root/build_artifacts/pyasn1-modules_1713209683338/work
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1711811537435/work
pydantic @ file:///home/conda/feedstock_root/build_artifacts/pydantic_1720293063581/work
pydantic_core @ file:///home/conda/feedstock_root/build_artifacts/pydantic-core_1720041200430/work
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1714846767233/work
PyJWT @ file:///home/conda/feedstock_root/build_artifacts/pyjwt_1706895065046/work
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1706660063483/work
pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1709721012883/work
PyQt5==5.15.9
PyQt5-sip==12.12.2
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1661604839144/work
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1709299778482/work
python-json-logger @ file:///home/conda/feedstock_root/build_artifacts/python-json-logger_1677079630776/work
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1706886791323/work
pyu2f @ file:///home/conda/feedstock_root/build_artifacts/pyu2f_1604248910016/work
PyYAML @ file:///home/conda/feedstock_root/build_artifacts/pyyaml_1695373611984/work
pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1715024370414/work
qtconsole @ file:///home/conda/feedstock_root/build_artifacts/qtconsole-base_1714942934316/work
QtPy @ file:///home/conda/feedstock_root/build_artifacts/qtpy_1698112029416/work
referencing @ file:///home/conda/feedstock_root/build_artifacts/referencing_1714619483868/work
regex==2024.7.24
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1717057054362/work
requests-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/requests-oauthlib_1711290127547/work
rfc3339-validator @ file:///home/conda/feedstock_root/build_artifacts/rfc3339-validator_1638811747357/work
rfc3986-validator @ file:///home/conda/feedstock_root/build_artifacts/rfc3986-validator_1598024191506/work
rich @ file:///home/conda/feedstock_root/build_artifacts/rich-split_1709150387247/work/dist
rpds-py @ file:///home/conda/feedstock_root/build_artifacts/rpds-py_1715090025935/work
rsa @ file:///home/conda/feedstock_root/build_artifacts/rsa_1658328885051/work
safetensors==0.4.4
scikit-learn @ file:///home/conda/feedstock_root/build_artifacts/scikit-learn_1719998107382/work/dist/scikit_learn-1.5.1-cp311-cp311-linux_x86_64.whl#sha256=149fab54f11e79470f037b956bb6061954107c264039bb95b7412ab36f06c972
scipy @ file:///home/conda/feedstock_root/build_artifacts/scipy-split_1720323007424/work/dist/scipy-1.14.0-cp311-cp311-linux_x86_64.whl#sha256=1555805d3d22eadcd79d8bbf4de2865c7ad881feceb57d3c2d91ec2469d4acf7
Send2Trash @ file:///home/conda/feedstock_root/build_artifacts/send2trash_1712584999685/work
sentencepiece==0.2.0
shellingham @ file:///home/conda/feedstock_root/build_artifacts/shellingham_1698144360966/work
simple_parsing==0.1.5
sip @ file:///home/conda/feedstock_root/build_artifacts/sip_1697300428897/work
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
smart_open @ file:///home/conda/feedstock_root/build_artifacts/smart_open_split_1711455333245/work/dist
sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1708952932303/work
soupsieve @ file:///home/conda/feedstock_root/build_artifacts/soupsieve_1693929250441/work
spacy @ file:///home/conda/feedstock_root/build_artifacts/spacy_1720075566465/work
spacy-legacy @ file:///home/conda/feedstock_root/build_artifacts/spacy-legacy_1674550301837/work
spacy-loggers @ file:///home/conda/feedstock_root/build_artifacts/spacy-loggers_1694527114282/work
SQLAlchemy==2.0.34
srsly @ file:///home/conda/feedstock_root/build_artifacts/srsly_1695654015181/work
stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work
sympy @ file:///home/conda/feedstock_root/build_artifacts/sympy_1718625539893/work
tenacity==8.5.0
tensorboard @ file:///home/conda/feedstock_root/build_artifacts/tensorboard_1695917943728/work/tensorboard-2.14.1-py3-none-any.whl#sha256=3db108fb58f023b6439880e177743c5f1e703e9eeb5fb7d597871f949f85fd58
tensorboard-data-server @ file:///home/conda/feedstock_root/build_artifacts/tensorboard-data-server_1695425375375/work/tensorboard_data_server-0.7.0-py3-none-manylinux2014_x86_64.whl#sha256=4a87e32f17958007f01c1acb90cf7aab5877e41b1a929e3a016020697c37b53d
tensorflow @ file:///home/conda/feedstock_root/build_artifacts/tensorflow-split_1699606989644/work/tensorflow_pkg/tensorflow-2.14.0-cp311-cp311-linux_x86_64.whl#sha256=581f1dc8aef4e7318d61e11cac8cefe9ab4066f8caedf02cbeadc93c00c569b9
tensorflow-estimator @ file:///home/conda/feedstock_root/build_artifacts/tensorflow-split_1699606989644/work/tensorflow-estimator/wheel_dir/tensorflow_estimator-2.14.0-py2.py3-none-any.whl#sha256=1aa91c08a19ee48a3fa84b8482235363e5f3dd5741d3373554049a40751b5735
termcolor @ file:///home/conda/feedstock_root/build_artifacts/termcolor_1704357939450/work
terminado @ file:///home/conda/feedstock_root/build_artifacts/terminado_1710262609923/work
thinc @ file:///home/conda/feedstock_root/build_artifacts/thinc_1715461672215/work
threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1714400101435/work
tiktoken==0.7.0
tinycss2 @ file:///home/conda/feedstock_root/build_artifacts/tinycss2_1713974937325/work
tokenizers==0.19.1
toml @ file:///home/conda/feedstock_root/build_artifacts/toml_1604308577558/work
tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1644342247877/work
torch==2.4.0
torchaudio==2.3.1
torchvision==0.18.1
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1717722848697/work
tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1714854870413/work
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1713535121073/work
transformers==4.44.2
triton==3.0.0
typer==0.12.3
typer-slim==0.12.3
types-python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/types-python-dateutil_1710589910274/work
typing-utils @ file:///home/conda/feedstock_root/build_artifacts/typing_utils_1622899189314/work
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1717802530399/work
tzdata @ file:///croot/python-tzdata_1690578112552/work
uri-template @ file:///home/conda/feedstock_root/build_artifacts/uri-template_1688655812972/work/dist
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1719391292974/work
wasabi @ file:///home/conda/feedstock_root/build_artifacts/wasabi_1715409627083/work
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1704731205417/work
weasel @ file:///home/conda/feedstock_root/build_artifacts/weasel_1719987001521/work
webcolors @ file:///home/conda/feedstock_root/build_artifacts/webcolors_1717667289718/work
webencodings @ file:///home/conda/feedstock_root/build_artifacts/webencodings_1694681268211/work
websocket-client @ file:///home/conda/feedstock_root/build_artifacts/websocket-client_1713923384721/work
Werkzeug @ file:///home/conda/feedstock_root/build_artifacts/werkzeug_1715000201436/work
widgetsnbextension @ file:///home/conda/feedstock_root/build_artifacts/widgetsnbextension_1716891659446/work
wrapt @ file:///home/conda/feedstock_root/build_artifacts/wrapt_1666806105113/work
xformers==0.0.27.post2
yarl @ file:///home/conda/feedstock_root/build_artifacts/yarl_1705508295175/work
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1718013267051/work
zstandard==0.22.0

Reproduction Steps

I am working on function calling methods, however i face challenged in the decode function as the output has not been consistent sometimes it produces the parameters required and other it errors as special_token_ policy error despite of turning it to ignore "1"

Here is my snippet, while use Nemo-Instruct-2407:

 model = Transformer.from_folder(setup.mistral_models_path)
 #tokenizer = MistralTokenizer.from_file(f"{setup.mistral_models_path}/tekken.json")
tokenizer = MistralTokenizer.v3(is_tekken=True)
tokenizer.special_token_policy = 1;
get_flow_definition = {
        "type": "function",
        "function": {
            "name": "get_flow",
            "description": "Get a flow using the id",
            "parameters": {
                   "type": "object",
                    "properties": {
                        "flow_id": {
                            "type": "string",
                            "description": "id of the flow",
                        },
                    },
                    "required": ["flow_id"],
                },
        },
    }
trial = [get_flow_definition]
readyTools = []
for tool in trial:
    if isinstance(tool, dict) and 'function' in tool:
        newTool =Tool(
            function = Function(
                name=tool["function"]["name"], 
                description=tool["function"]["description"],
                parameters=tool["function"]["parameters"]
            )
        )
        readyTools.append(newTool)

print(readyTools)

prompt = "can you get me a flow?"

messges = [
    UserMessage(content=prompt)
]
completion_request = ChatCompletionRequest(
    tools=readyTools,
    messages=[UserMessage(content=prompt)],
)

tokens = tokenizer.encode_chat_completion(completion_request).tokens
out_tokens, _ = generate([tokens], model, max_tokens=1024, temperature=0.35, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
result = tokenizer.decode(out_tokens[0])
print(result)

My issue due to output creates fluctuation, most of the time produces the below error:

`---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[4], line 48
     46 out_tokens, _ = generate([tokens], model, max_tokens=1024, temperature=0.35, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
     47 tokenizer.special_token_policy = 1;
---> 48 result = tokenizer.decode(out_tokens[0])
     49 print(result)

File [~/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/mistral.py:148](http://147.185.40.32:20045/lab/tree/ai-worker/src/v1/ai/model/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/mistral.py#line=147), in MistralTokenizer.decode(self, tokens)
    147 def decode(self, tokens: List[int]) -> str:
--> 148     return self.instruct_tokenizer.decode(tokens)

File [~/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/sentencepiece.py:200](http://147.185.40.32:20045/lab/tree/ai-worker/src/v1/ai/model/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/sentencepiece.py#line=199), in InstructTokenizerBase.decode(self, tokens)
    199 def decode(self, tokens: List[int]) -> str:
--> 200     return self.tokenizer.decode(tokens)

File [~/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/tekken.py:234](http://147.185.40.32:20045/lab/tree/ai-worker/src/v1/ai/model/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/tekken.py#line=233), in Tekkenizer.decode(self, tokens)
    233 def decode(self, tokens: List[int]) -> str:
--> 234     return "".join(self._decode_all(tokens, special_token_policy=self._special_token_policy))

File [~/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/tekken.py:203](http://147.185.40.32:20045/lab/tree/ai-worker/src/v1/ai/model/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/tekken.py#line=202), in Tekkenizer._decode_all(self, tokens, special_token_policy)
    201 if is_special:
    202     if special_token_policy == SpecialTokenPolicy.RAISE:
--> 203         raise ValueError(
    204             f"Decoding `tokens` that contain special tokens ({list(group)}) is not allowed. \n"
    205             "Either make sure `tokens` do not include any special tokens or, "
    206             "if you want to decode `tokens` that includes special tokens, "
    207             "change the tokenizer's special token policy to IGNORE or KEEP: \n"
    208             "```\nfrom mistral_common.tokens.tokenizers.mistral import MistralTokenizer"
    209             "\nfrom mistral_common.tokens.tokenizers.tekken import SpecialTokenPolicy"
    210             "\n\ntokenizer = MistralTokenizer.v3(is_tekken=True)"
    211             "\ntokenizer.special_token_policy = SpecialTokenPolicy.IGNORE  # or SpecialTokenPolicy.KEEP"
    212             "\n```"
    213         )
    214     elif special_token_policy == SpecialTokenPolicy.KEEP:
    215         decoded.extend(self._all_special_tokens[t] for t in group)

ValueError: Decoding `tokens` that contain special tokens ([9]) is not allowed. 
Either make sure `tokens` do not include any special tokens or, if you want to decode `tokens` that includes special tokens, change the tokenizer's special token policy to IGNORE or KEEP: 

from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_common.tokens.tokenizers.tekken import SpecialTokenPolicy

tokenizer = MistralTokenizer.v3(is_tekken=True)
tokenizer.special_token_policy = SpecialTokenPolicy.IGNORE # or SpecialTokenPolicy.KEEP

```

### Expected Behavior

Expected is to produce the params of the flow id required

### Additional Context

_No response_

### Suggested Solutions

_No response_

[Mohrizk]

It consistently worked when i manually removed the special token causes this issue [9] with the below line
filtered_tokens = [token for token in out_tokens[0] if token != 9]

I still think we need a valid solution for this, if possible

[patrickvonplaten]

Hey @Mohrizk,

Thanks for the issue - can you please make sure to post a fully, reproducible code snippet that I can copy-paste into a Python shell and it will run correctly.

For the above code snippet, I don't know exactly where you took the model weights from. Also Transformer and MistralTokenizer are not imported so the code snippet is not runable.

Can you try to post a complete code snippet please?