Skip to content

Commit

Permalink
Revert "Revert "Tutorial experiment update (#569)" (#570)"
Browse files Browse the repository at this point in the history 
This reverts commit fe6ab55.
  • Loading branch information
@ngrayluna
ngrayluna authored Aug 22, 2024
1 parent fe6ab55 commit 19d9320
Showing 1 changed file with 75 additions and 69 deletions.
144 changes: 75 additions & 69 deletions colabs/intro/Intro_to_Weights_&_Biases.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,44 +20,18 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# 🏃‍♀️ Quickstart\n",
"Use **[Weights & Biases](https://wandb.ai/site?utm_source=intro_colab&utm_medium=code&utm_campaign=intro)** for machine learning experiment tracking, model checkpointing, and collaboration with your team. See the full Weights & Biases Documentation **[here](https://docs.wandb.ai/quickstart)**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🤩 A shared dashboard for your experiments\n",
"Use [W&B](https://wandb.ai/site?utm_source=intro_colab&utm_medium=code&utm_campaign=intro) for machine learning experiment tracking, model checkpointing, collaboration with your team and more. See the full W&B Documentation [here](https://docs.wandb.ai/).\n",
"\n",
"With just a few lines of code,\n",
"you'll get rich, interactive, shareable dashboards [which you can see yourself here](https://wandb.ai/wandb/wandb_example).\n",
"![](https://i.imgur.com/Pell4Oo.png)"
"In this notebook, you will create and track a machine learning experiment using a simple PyTorch model. By the end of the notebook, you will have an interactive project dashboard that you can share and customize with other members of your team. [View an example dashboard here](https://wandb.ai/wandb/wandb_example)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"## 🔒 Data & Privacy\n",
"\n",
"We take security very seriously, and our cloud-hosted dashboard uses industry standard best practices for encryption. If you're working with models and datasets that cannot leave your enterprise cluster, we have [on-prem](https://docs.wandb.com/self-hosted) installations available.\n",
"\n",
"It's also easy to download all your data and export it to other tools — like custom analysis in a Jupyter notebook. Here's [more on our API](https://docs.wandb.com/library/api).\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🪄 Install `wandb` library and login\n",
"\n",
"\n",
"Start by installing the library and logging in to your free account.\n",
"\n"
"Install the W&B Python SDK and log in:"
]
},
{
Expand All @@ -77,6 +51,8 @@
"source": [
"# Log in to your W&B account\n",
"import wandb\n",
"import random\n",
"import math\n",
"\n",
"# Use wandb-core, temporary for wandb's new backend \n",
"wandb.require(\"core\")"
Expand All @@ -95,12 +71,12 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## 👟 Run an experiment\n",
"1️⃣. **Start a new run** and pass in hyperparameters to track\n",
"## Simulate and track a machine learning experiment with W&B\n",
"\n",
"2️⃣. **Log metrics** from training or evaluation\n",
"Create, track, and visualize a machine learning experiment. To do this:\n",
"\n",
"3️⃣. **Visualize results** in the dashboard"
"1. Initialize a [W&B run](https://docs.wandb.ai/guides/runs) and pass in the hyperparameters you want to track.\n",
"2. Within your training loop, log metrics such as the accuruacy and loss."
]
},
{
Expand All @@ -115,7 +91,7 @@
"# Launch 5 simulated experiments\n",
"total_runs = 5\n",
"for run in range(total_runs):\n",
" # 🐝 1️⃣ Start a new run to track this script\n",
" # 1️. Start a new run to track this script\n",
" wandb.init(\n",
" # Set the project where this run will be logged\n",
" project=\"basic-intro\",\n",
Expand All @@ -136,7 +112,7 @@
" acc = 1 - 2 ** -epoch - random.random() / epoch - offset\n",
" loss = 2 ** -epoch + random.random() / epoch + offset\n",
"\n",
" # 🐝 2️⃣ Log metrics from your script to W&B\n",
" # 2️. Log metrics from your script to W&B\n",
" wandb.log({\"acc\": acc, \"loss\": loss})\n",
"\n",
" # Mark the run as finished\n",
Expand All @@ -147,33 +123,37 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"3️⃣ You can find your interactive dashboard by clicking any of the 👆 wandb links above."
"View how your machine learning peformed in your W&B project. Copy and paste the URL link that is printed from the previous cell. The URL will redirect you to a W&B project that contains a dashboard showing graphs the show how \n",
"\n",
"The following image shows what a dashboard can look like:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](https://i.imgur.com/Pell4Oo.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 🔥 Simple Pytorch Neural Network\n",
"Now that we know how to integrate W&B into a psuedo machine learning training loop, let's track a machine learning experiment using a basic PyTorch neural network. The following code will also upload model checkpoints to W&B that you can then share with other teams in in your organization.\n",
"\n",
"💪 Run this model to train a simple MNIST classifier, and click on the project page link to see your results stream in live to a W&B project.\n",
"## Track a machine learning experiment using Pytorch\n",
"\n",
"The following code cell defines and trains a simple MNIST classifier. During training, you will see W&B prints out URLs. Click on the project page link to see your results stream in live to a W&B project.\n",
"\n",
"Any run in `wandb` automatically logs [metrics](https://docs.wandb.ai/ref/app/pages/run-page#charts-tab),\n",
"W&B runs automatically log [metrics](https://docs.wandb.ai/ref/app/pages/run-page#charts-tab),\n",
"[system information](https://docs.wandb.ai/ref/app/pages/run-page#system-tab),\n",
"[hyperparameters](https://docs.wandb.ai/ref/app/pages/run-page#overview-tab),\n",
"[terminal output](https://docs.wandb.ai/ref/app/pages/run-page#logs-tab) and\n",
"you'll see an [interactive table](https://docs.wandb.ai/guides/data-vis)\n",
"with model inputs and outputs. \n",
"\n",
"This code will also save and upload Model checkpoints to W&B where you can then use them in the [W&B Registry](https://docs.wandb.ai/guides/registry)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set up Dataloader"
"### Set up PyTorch Dataloader\n",
"The following cell defines some useful functions that we will need to train our machine learning model. The functions themselves are not unique to W&B so we'll not cover them in detail here. See the PyTorch documentation for more information on how to define [forward and backward training loop](https://pytorch.org/tutorials/beginner/nn_tutorial.html), how to use [PyTorch DataLoaders](https://pytorch.org/tutorials/beginner/basics/data_tutorial.html) to load data in for training, and how define PyTorch models using the [`torch.nn.Sequential` Class](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html). "
]
},
{
Expand Down Expand Up @@ -232,11 +212,31 @@
" # Log one batch of images to the dashboard, always same batch_idx.\n",
" if i==batch_idx and log_images:\n",
" log_image_table(images, predicted, labels, outputs.softmax(dim=1))\n",
" return val_loss / len(valid_dl.dataset), correct / len(valid_dl.dataset)\n",
" return val_loss / len(valid_dl.dataset), correct / len(valid_dl.dataset)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create a teble to compare the predicted values versus the true value\n",
"\n",
"The following cell is unique to W&B, so let's go over it.\n",
"\n",
"In the cell we define a function called `log_image_table`. Though technically, optional, this function creates a W&B Table object. We will use the table object to create a table that shows what the model predicted for each image. \n",
"\n",
"More specifically, each row will conists of the image fed to the model, along with predicted value and the actual value (label). "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def log_image_table(images, predicted, labels, probs):\n",
" \"Log a wandb.Table with (img, pred, target, scores)\"\n",
" # 🐝 Create a wandb Table to log images, labels and predictions to\n",
" # Create a wandb Table to log images, labels and predictions to\n",
" table = wandb.Table(columns=[\"image\", \"pred\", \"target\"]+[f\"score_{i}\" for i in range(10)])\n",
" for img, pred, targ, prob in zip(images.to(\"cpu\"), predicted.to(\"cpu\"), labels.to(\"cpu\"), probs.to(\"cpu\")):\n",
" table.add_data(wandb.Image(img[0].numpy()*255), pred, targ, *prob.numpy())\n",
Expand All @@ -247,7 +247,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Train Your Model & Upload Checkpoints"
"### Train your model and upload checkpoints\n",
"\n",
"The following code trains and saves model checkpoints to your project. Use model checkpoints like you normally would to assess how the model performed during training. \n",
"\n",
"W&B also makes it easy to share your saved models and model checkpoints with other members of your team or organization. To learn how to share your model and model checkpoints with members outside of your team, see [W&B Registry](https://docs.wandb.ai/guides/registry)."
]
},
{
Expand All @@ -258,7 +262,7 @@
"source": [
"# Launch 3 experiments, trying different dropout rates\n",
"for _ in range(3):\n",
" # 🐝 initialise a wandb run\n",
" # initialise a wandb run\n",
" wandb.init(\n",
" project=\"pytorch-intro\",\n",
" config={\n",
Expand Down Expand Up @@ -303,49 +307,50 @@
" \"train/example_ct\": example_ct}\n",
"\n",
" if step + 1 < n_steps_per_epoch:\n",
" # 🐝 Log train metrics to wandb\n",
" # Log train metrics to wandb\n",
" wandb.log(metrics)\n",
"\n",
" step_ct += 1\n",
"\n",
" val_loss, accuracy = validate_model(model, valid_dl, loss_func, log_images=(epoch==(config.epochs-1)))\n",
"\n",
" # 🐝 Log train and validation metrics to wandb\n",
" # Log train and validation metrics to wandb\n",
" val_metrics = {\"val/val_loss\": val_loss,\n",
" \"val/val_accuracy\": accuracy}\n",
" wandb.log({**metrics, **val_metrics})\n",
"\n",
" # 🐝 Save the model checkpoint to wandb\n",
" # Save the model checkpoint to wandb\n",
" torch.save(model, \"my_model.pt\")\n",
" wandb.log_model(\"./my_model.pt\", \"my_mnist_model\", aliases=[f\"epoch-{epoch+1}_dropout-{round(wandb.config.dropout, 4)}\"])\n",
"\n",
" print(f\"Epoch: {epoch+1}, Train Loss: {train_loss:.3f}, Valid Loss: {val_loss:3f}, Accuracy: {accuracy:.2f}\")\n",
"\n",
" # 🐝 If you had a test set, this is how you could log it as a Summary metric\n",
" # If you had a test set, this is how you could log it as a Summary metric\n",
" wandb.summary['test_accuracy'] = 0.8\n",
"\n",
" # 🐝 Close your wandb run\n",
" # Close your wandb run\n",
" wandb.finish()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You have now trained your first model using wandb! 👆 Click on the wandb link above to see your metrics and see your saved Model checkpoints in the Artifacts tab in the UI"
"You have now trained your first model using W&B. Click on one of the links above to see your metrics and see your saved model checkpoints in the Artifacts tab in the W&B App UI"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 🔔 Try W&B Alerts\n",
"## (Optional) Set up a W&B Alert\n",
"\n",
"**[W&B Alerts](https://docs.wandb.ai/guides/track/alert)** allows you to send alerts, triggered from your Python code, to your Slack or email. There are 2 steps to follow the first time you'd like to send a Slack or email alert, triggered from your code:\n",
"Create a [W&B Alerts](https://docs.wandb.ai/guides/track/alert) to send alerts to your Slack or email from your Python code. \n",
"\n",
"1) Turn on Alerts in your W&B [User Settings](https://wandb.ai/settings)\n",
"There are 2 steps to follow the first time you'd like to send a Slack or email alert, triggered from your code:\n",
"\n",
"2) Add `wandb.alert()` to your code:\n",
"1) Turn on Alerts in your W&B [User Settings](https://wandb.ai/settings)\n",
"2) Add `wandb.alert()` to your code. For example:\n",
"\n",
"```python\n",
"wandb.alert(\n",
Expand All @@ -354,7 +359,7 @@
")\n",
"```\n",
"\n",
"See the minimal example below to see how to use `wandb.alert`. You can find the full docs for **[W&B Alerts here](https://docs.wandb.ai/guides/track/alert)**"
"The following cell shows a minimal example below to see how to use `wandb.alert`"
]
},
{
Expand All @@ -374,12 +379,12 @@
" accuracy = round(random.random() + random.random(), 3)\n",
" print(f'Accuracy is: {accuracy}, {acc_threshold}')\n",
"\n",
" # 🐝 Log accuracy to wandb\n",
" # Log accuracy to wandb\n",
" wandb.log({\"Accuracy\": accuracy})\n",
"\n",
" # 🔔 If the accuracy is below the threshold, fire a W&B Alert and stop the run\n",
" # If the accuracy is below the threshold, fire a W&B Alert and stop the run\n",
" if accuracy <= acc_threshold:\n",
" # 🐝 Send the wandb Alert\n",
" # Send the wandb Alert\n",
" wandb.alert(\n",
" title='Low Accuracy',\n",
" text=f'Accuracy {accuracy} at step {training_step} is below the acceptable theshold, {acc_threshold}',\n",
Expand All @@ -395,10 +400,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"You can find the full docs for [W&B Alerts here](https://docs.wandb.ai/guides/track/alert).\n",
"\n",
"# What's next 🚀 ?\n",
"## Next steps\n",
"The next tutorial you will learn how to do hyperparameter optimization using W&B Sweeps:\n",
"## 👉 [Hyperparameters sweeps using PyTorch](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/pytorch/Organizing_Hyperparameter_Sweeps_in_PyTorch_with_W%26B.ipynb)"
"[Hyperparameters sweeps using PyTorch](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/pytorch/Organizing_Hyperparameter_Sweeps_in_PyTorch_with_W%26B.ipynb)"
]
}
],
Expand All @@ -415,5 +421,5 @@
}
},
"nbformat": 4,
"nbformat_minor": 0
"nbformat_minor": 4
}

0 comments on commit 19d9320

Please sign in to comment.