How to clean up /tmp/python_env_XXXXXX when using python backend?

[patryk126p]

Hi, I'm using Triton to serve multiple models using python backend. I use official 23.08-pyt-python-py3 image, my model repository resides on GCS and models are loaded and unloaded quite regularly using mode=EXPLICIT. All models use additional python libraries packed with conda-pack into .tar.gz archives
With time I've started noticing that my free disc space is shrinking the more models I load/unload (number of models loaded at a time being ~constant). My investigation led me to /tmp dir. Its contents look like this:

# ls /tmp
folderE12rEy  folderMY4q9Z  folderNJ5E9t  folderWHaZQG  folderbGn41Q  foldernZeisU  lost+found
folderLh4Hrz  folderNEVdM0  folderVx4z0F  folderb4WT8e  folderl0Da2U  folderxT3fqx  python_env_z0agTf

Every folderXXXXXX corresponds to one currently loaded model and such directories are deleted once the model is unloaded - all well and good. But it looks like that's not the case for contents of python_env_XXXXXX which seems to accumulate with time:

# ls /tmp/python_env_z0agTf/
0  10  12  14  16  18  2   21  23  25  27  29  30  32  34  36  38  5  7  9
1  11  13  15  17  19  20  22  24  26  28  3   31  33  35  37  4   6  8

What I would like to know is how can I clean up /tmp/python_env_XXXXXX to free up the space?
That could be broken up into two sub questions:

• is there a way to tell Triton to do that automatically (maybe via finalize method of TritonPythonModel class)?
• how can I know which subdirectories of /tmp/python_env_XXXXXX can be safely deleted?

[oandreeva-nv]

Interesting, this seem to be a bug, thanks for reporting this. I'll file an issue for the team.

[oandreeva-nv]

For internal reference: issue 5957

[maisondono]

@oandreeva-nv Has this issue been fixed? We are still encountered the same issue with triton 24.05.