Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ludwig upload to push artifacts to HuggingFace Hub #3480

Merged
merged 10 commits into from
Jul 29, 2023
Merged

Conversation

arnavgarg1
Copy link
Contributor

@arnavgarg1 arnavgarg1 commented Jul 25, 2023

This PR adds a new command called ludwig upload to push trained model artifacts to Huggingface Hub.

The API is very simple:

ludwig upload hf_hub --repo_id <repo_id> --model_path </path/to/saved/model>

When you run this command, it will check if you are already signed into Huggingface Hub. This can be done through huggingface-cli login in advance, otherwise it will make you follow the steps to sign in which looks something like this:

$ ludwig upload --service hf_hub -r arnavgrg/test -m /workspaces/trained_models/api_experiment_run

    _|    _|  _|    _|    _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|_|_|_|    _|_|      _|_|_|  _|_|_|_|
    _|    _|  _|    _|  _|        _|          _|    _|_|    _|  _|            _|        _|    _|  _|        _|
    _|_|_|_|  _|    _|  _|  _|_|  _|  _|_|    _|    _|  _|  _|  _|  _|_|      _|_|_|    _|_|_|_|  _|        _|_|_|
    _|    _|  _|    _|  _|    _|  _|    _|    _|    _|    _|_|  _|    _|      _|        _|    _|  _|        _|
    _|    _|    _|_|      _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|        _|    _|    _|_|_|  _|_|_|_|
    
    To login, `huggingface_hub` requires a token generated from https://huggingface.co/settings/tokens .
Token: 
Add token as git credential? (Y/n) Y
Token is valid (permission: write).
Your token has been saved in your configured git credential helpers (!f()).
Your token has been saved to /home/vscode/.cache/huggingface/token
Login successful

You can find your token at https://huggingface.co/settings/tokens.

Once you're signed in, this will create a new public model repository on Huggingface Hub called repo_id if it doesn't exist, and push the trained model weights and associated configs in model_path to repo_id.

There are some other optional parameters you can pass to ludwig upload:

  1. private: Whether to create a private repository or a public repository.
  2. commit_message: The summary / title / first line of the generated commit.
  3. commit_description: The description of the generated commit.

@github-actions
Copy link

github-actions bot commented Jul 25, 2023

Unit Test Results

       6 files  ±       0         6 suites  ±0   1h 14m 58s ⏱️ + 11m 47s
2 775 tests +2 741  2 756 ✔️ +2 727  12 💤 +7  7 +7 
2 818 runs  +2 730  2 790 ✔️ +2 718  21 💤 +5  7 +7 

For more details on these failures, see this check.

Results for commit 1d11569. ± Comparison against base commit 62a5524.

♻️ This comment has been updated with latest results.

ludwig/commit.py Outdated Show resolved Hide resolved
ludwig/utils/commit_utils.py Outdated Show resolved Hide resolved
ludwig/utils/commit_utils.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@tgaddair tgaddair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome!

I'm kind of torn on the naming. I think commit is somewhat counterintuitive to me. The HF Hub API itself uses the terminology "upload" which feels more clear. Maybe we can do that instead? I also think making it clear we're uploading to HF (as opposed to Predibase or MLFlow or whatever) is important.

So to that end I would recommend the following:

ludwig upload hf_hub ...

What do you think?

ludwig/cli.py Outdated Show resolved Hide resolved
)

# Make sure the model's saved artifacts either contain:
# 1. pytorch_model.bin -> regular model training, such as ECD or for LLMs
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does HF support non-HF models being uploaded to the hub?

ludwig/commit.py Outdated Show resolved Hide resolved
@arnavgarg1 arnavgarg1 changed the title Add ludwig commit to push artifacts to HuggingFace Hub Add ludwig upload to push artifacts to HuggingFace Hub Jul 29, 2023
@arnavgarg1 arnavgarg1 merged commit f1ad0df into master Jul 29, 2023
13 of 15 checks passed
@arnavgarg1 arnavgarg1 deleted the push_to_hub branch July 29, 2023 22:03
ludwig/utils/upload_utils.py Show resolved Hide resolved
ludwig/api.py Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants