Skip to content

Commit

Permalink
Update deprecated HuggingFace function (#5144)
Browse files Browse the repository at this point in the history
Changing `list_files_info` to `list_repo_tree`:

```
FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
```
  • Loading branch information
mrwyattii authored Feb 16, 2024
1 parent 177dc14 commit a37e59b
Showing 1 changed file with 2 additions and 5 deletions.
7 changes: 2 additions & 5 deletions deepspeed/inference/v2/checkpoint/huggingface_engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,16 +40,13 @@ def _fetch_checkpoint_files(self):
# currently coming from the ckpt engine init but maybe a catch all kwargs for other
# snapshot download parameters would be more flexible.

# NOTE(jeff): allow_patterns here are explicitly not using safetensors or other
# checkpoint files that may be present. Example of all files in the llama-2-7b
# repo here: https://huggingface.co/meta-llama/Llama-2-7b-hf/tree/main
from huggingface_hub import snapshot_download, list_files_info
from huggingface_hub import snapshot_download, list_repo_tree

def model_has_safetensors(model_name_or_path: str) -> bool:
if os.path.isdir(model_name_or_path):
file_list = os.listdir(model_name_or_path)
else:
file_list = [rf.rfilename for rf in list_files_info(model_name_or_path)]
file_list = [rf.path for rf in list_repo_tree(model_name_or_path)]
for f in file_list:
if f.endswith(".safetensors"):
return True
Expand Down

0 comments on commit a37e59b

Please sign in to comment.