Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add model artifact and registry info to huggingface examples #451

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

feat: add model artifact and registry info to huggingface examples #451

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
85a61e0
feat: add model artifact and registry info to huggingface glue example
Aug 18, 2023
b659a0c
fix: change model artifacts example and add artifacts workspace
Aug 22, 2023
5409a95
update: add model checkpoint logging and model registry content to Op…
Aug 22, 2023
23a382e
update: add model registry and checkpoint to visualize hf examples
Aug 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 37 additions & 22 deletions colabs/huggingface/Huggingface_wandb.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -10,18 +9,16 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Hugging Face + W&B\n",
"Visualize your [Hugging Face](https://github.com/huggingface/transformers) model's performance quickly with a seamless [W&B](https://wandb.ai/site) integration.\n",
"\n",
"Compare hyperparameters, output metrics, and system stats like GPU utilization across your models. \n"
"Compare hyperparameters, output metrics, and system stats like GPU utilization across your models.\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -31,7 +28,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -52,7 +48,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -63,15 +58,13 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# 🚀 Install, Import, and Log in\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand Down Expand Up @@ -104,15 +97,13 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🖊️ [Sign up for a free account →](https://app.wandb.ai/login?signup=true)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -131,7 +122,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -144,12 +134,11 @@
"metadata": {},
"outputs": [],
"source": [
"# Optional: log both gradients and parameters\n",
"%env WANDB_WATCH=all"
"# Optional: log model checkpoints as W&B Artifacts\n",
"%env WANDB_LOG_MODEL=\"checkpoint\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -164,6 +153,7 @@
"outputs": [],
"source": [
"%env WANDB_PROJECT=huggingface-demo\n",
"%env WANDB_LOG_MODEL=checkpoint\n",
"%env TASK_NAME=MRPC\n",
"\n",
"!python run_glue.py \\\n",
Expand All @@ -177,11 +167,13 @@
" --num_train_epochs 3 \\\n",
" --output_dir /tmp/$TASK_NAME/ \\\n",
" --overwrite_output_dir \\\n",
" --logging_steps 50"
" --logging_steps 50 \\\n",
" --save_steps 100 \\\n",
" --eval_steps 100 \\\n",
" --report_to \"wandb\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -199,12 +191,37 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model Artifacts\n",
"\n",
"### 📈 Track key information effortlessly by default\n",
"See an example model checkpoint and its in the UI [here](https://wandb.ai/wandb/arttest/artifacts/model/iv3_trained/5334ab69740f9dda4fed/lineage?_gl=1*yyql5q*_ga*MTQxOTYyNzExOS4xNjg0NDYyNzk1*_ga_JH1SJHJQXJ*MTY5MjMwNzI2Mi4yNjkuMS4xNjkyMzA5NjM2LjM3LjAuMA..) These artifacts store all the metadata associated with this model, the W&B runs consuming it, and the whole lineage of upstream and downstream artifacts!\n",
"\n",
"[Here's](https://wandb.ai/parambharat/huggingface-demo/runs/szfbcsd7/artifacts?workspace=) an example of the artifacts generated from the above run\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model Registry\n",
"After logging a bunch of checkpoints across multiple runs during experimentation, now comes time to hand-off the best checkpoint to the next stage of the workflow (e.g. testing, deployment).\n",
"\n",
"The Model Registry is a central page that lives above individual W&B projects. It houses **Registered Models**, portfolios that store \"links\" to the valuable checkpoints living in individual W&B Projects.\n",
"\n",
"The model registry offers a centralized place to house the best checkpoints for all your model tasks. Any `model` artifact you log can be \"linked\" to a Registered Model. Pleaser refer to [this](https://docs.wandb.ai/guides/models) guide where you can find a [walkthrough](https://docs.wandb.ai/guides/models/walkthrough) of the Model Registry to learn more."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## 📈 Track key information effortlessly by default\n",
"Weights & Biases saves a new run for each experiment. Here's the information that gets saved by default:\n",
"- **Hyperparameters**: Settings for your model are saved in Config\n",
"- **Model Metrics**: Time series data of metrics streaming in are saved in Log\n",
Expand All @@ -213,21 +230,19 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🤓 Learn more!\n",
"- [Documentation](https://docs.wandb.com/huggingface): docs on the Weights & Biases and Hugging Face integration\n",
"- [Videos](http://wandb.me/youtube): tutorials, interviews with practitioners, and more on our YouTube channel\n",
"- Contact: Message us at [email protected] with questions "
"- Contact: Message us at [email protected] with questions"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"include_colab_link": true,
"provenance": [],
"toc_visible": true
},
Expand All @@ -237,5 +252,5 @@
}
},
"nbformat": 4,
"nbformat_minor": 4
"nbformat_minor": 0
}
11 changes: 8 additions & 3 deletions colabs/huggingface/Optimize_Hugging_Face_models_with_Weights_&_Biases.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@
"metadata": {},
"outputs": [],
"source": [
"%env WANDB_LOG_MODEL=true"
"%env WANDB_LOG_MODEL=\"checkpoint\""
]
},
{
Expand Down Expand Up @@ -742,7 +742,12 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Once you're happy with a model, don't forget to [share it with the word](https://huggingface.co/transformers/model_sharing.html) on the Model Hub!"
"## Model Registry\n",
"After logging a bunch of checkpoints across multiple runs during experimentation, now comes time to hand-off the best checkpoint to the next stage of the workflow (e.g. testing, deployment).\n",
"\n",
"The Model Registry is a central page that lives above individual W&B projects. It houses **Registered Models**, portfolios that store \"links\" to the valuable checkpoints living in individual W&B Projects.\n",
"\n",
"The model registry offers a centralized place to house the best checkpoints for all your model tasks. Any `model` artifact you log can be \"linked\" to a Registered Model. Pleaser refer to [this](https://docs.wandb.ai/guides/models) guide where you can find a [walkthrough](https://docs.wandb.ai/guides/models/walkthrough) of the Model Registry to learn more."
]
},
{
Expand All @@ -761,7 +766,7 @@
"source": [
"## ❓ Questions about W&B\n",
"\n",
"If you have any questions about using W&B to track your model performance and predictions, please reach out to the [slack community](http://bit.ly/wandb-forum)."
"If you have any questions about using W&B to track your model performance and predictions, please reach out to the [discord community](https://wandb.me/discord)."
]
}
],
Expand Down
35 changes: 33 additions & 2 deletions colabs/huggingface/Visualize_your_Hugging_Face_data_with_Weights_&_Biases.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -419,8 +419,8 @@
"metadata": {},
"outputs": [],
"source": [
"# automatically log model to W&B at the end\n",
"%env WANDB_LOG_MODEL=true"
"# automatically log model checkpoints to W&B\n",
"%env WANDB_LOG_MODEL=\"checkpoint\""
]
},
{
Expand Down Expand Up @@ -537,6 +537,37 @@
"source": [
"wandb.finish()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model Registry\n",
"After logging a bunch of checkpoints across multiple runs during experimentation, now comes time to hand-off the best checkpoint to the next stage of the workflow (e.g. testing, deployment).\n",
"\n",
"The Model Registry is a central page that lives above individual W&B projects. It houses **Registered Models**, portfolios that store \"links\" to the valuable checkpoints living in individual W&B Projects.\n",
"\n",
"The model registry offers a centralized place to house the best checkpoints for all your model tasks. Any `model` artifact you log can be \"linked\" to a Registered Model. Pleaser refer to [this](https://docs.wandb.ai/guides/models) guide where you can find a [walkthrough](https://docs.wandb.ai/guides/models/walkthrough) of the Model Registry to learn more."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 📚 Resources\n",
"\n",
"* [Hugging Face and W&B integration documentation](https://docs.wandb.ai/integrations/huggingface) contains a few tips for taking most advantage of W&B\n",
"* [🤗 Transformers documentation](https://huggingface.co/transformers/) is extremely thorough and full of examples"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## ❓ Questions about W&B\n",
"\n",
"If you have any questions about using W&B to track your model performance and predictions, please reach out to the [discord community](https://wandb.me/discord)."
]
}
],
"metadata": {
Expand Down