-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ludwig upload
to push artifacts to HuggingFace Hub
#3480
Conversation
Unit Test Results 6 files ± 0 6 suites ±0 1h 14m 58s ⏱️ + 11m 47s For more details on these failures, see this check. Results for commit 1d11569. ± Comparison against base commit 62a5524. ♻️ This comment has been updated with latest results. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is awesome!
I'm kind of torn on the naming. I think commit
is somewhat counterintuitive to me. The HF Hub API itself uses the terminology "upload" which feels more clear. Maybe we can do that instead? I also think making it clear we're uploading to HF (as opposed to Predibase or MLFlow or whatever) is important.
So to that end I would recommend the following:
ludwig upload hf_hub ...
What do you think?
ludwig/utils/commit_utils.py
Outdated
) | ||
|
||
# Make sure the model's saved artifacts either contain: | ||
# 1. pytorch_model.bin -> regular model training, such as ECD or for LLMs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does HF support non-HF models being uploaded to the hub?
ludwig commit
to push artifacts to HuggingFace Hubludwig upload
to push artifacts to HuggingFace Hub
This PR adds a new command called
ludwig upload
to push trained model artifacts to Huggingface Hub.The API is very simple:
When you run this command, it will check if you are already signed into Huggingface Hub. This can be done through
huggingface-cli login
in advance, otherwise it will make you follow the steps to sign in which looks something like this:You can find your token at
https://huggingface.co/settings/tokens
.Once you're signed in, this will create a new public model repository on Huggingface Hub called
repo_id
if it doesn't exist, and push the trained model weights and associated configs inmodel_path
torepo_id
.There are some other optional parameters you can pass to
ludwig upload
:private
: Whether to create a private repository or a public repository.commit_message
: The summary / title / first line of the generated commit.commit_description
: The description of the generated commit.