diff --git a/colabs/huggingface/Visualize_your_Hugging_Face_data_with_Weights_&_Biases.ipynb b/colabs/huggingface/Visualize_your_Hugging_Face_data_with_Weights_&_Biases.ipynb
deleted file mode 100644
index 529ce653..00000000
--- a/colabs/huggingface/Visualize_your_Hugging_Face_data_with_Weights_&_Biases.ipynb
+++ /dev/null
@@ -1,564 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\")
\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "![](\"https://i.imgur.com/vnejHGh.png\")
\n",
- "\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Visualize your π€ Hugging Face data with πββοΈ Weights & Biases "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "[Hugging Face](https://huggingface.co/) provides tools to quickly train neural networks for NLP (Natural Language Processing) on any task (classification, translation, question answering, etc) and any dataset with PyTorch and TensorFlow 2.0.\n",
- "\n",
- "Coupled with [Weights & Biases integration](https://docs.wandb.ai/integrations/huggingface), you can quickly train and monitor models for full traceability and reproducibility without any extra line of code! You just need to install the library, sign in, and your experiments will automatically be logged:\n",
- "\n",
- "```bash\n",
- "pip install wandb\n",
- "wandb login\n",
- "```\n",
- "\n",
- "**Note**: To enable logging to W&B, set `report_to` to `wandb` in your `TrainingArguments` or script."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## π οΈ Installation and set-up"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "!pip install -qU transformers datasets wandb"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import wandb"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "wandb.login()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# π« Data and model preparation"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### π·οΈ Loading a dataset"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We'll download the\n",
- "[Yahoo! Answers Dataset](https://paperswithcode.com/sota/text-classification-on-yahoo-answers),\n",
- "which includes questions and answers from the now-defunct\n",
- "[Yahoo! Answers forum](https://en.wikipedia.org/wiki/Yahoo!_Answers)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from datasets import load_dataset\n",
- "dataset = load_dataset(\"yahoo_answers_topics\")\n",
- "dataset"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Data includes question titles, question bodies and best answers (for some questions), and a question topic from one of 10 choices."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "dataset['train'][0]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "For our text classification task, we use `question_title` as input and try to predict `topic`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "label_list = dataset['train'].unique('topic')\n",
- "label_list.sort()\n",
- "label_list"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "num_labels = len(label_list)\n",
- "num_labels"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The \"topic\" class needs to be renamed to \"labels\" for the `Trainer` to find it."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "dataset = dataset.rename_column('topic', 'labels')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "For the purpose of this demo, we'll decrease the size of the dataset for faster training."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "dataset['train'] = dataset['train'].select(range(dataset['train'].num_rows // 10))\n",
- "dataset['test'] = dataset['test'].select(range(dataset['test'].num_rows // 10))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### βοΈ Tokenizing the dataset"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "In a typical NLP workflow, we must first tokenize our dataset -- converting the stream of characters in the text into a stream of defined \"tokens\", which can be anything from a smaller set of characters to words from a vocabulary.\n",
- "\n",
- "We will use a pretrained model for our , so we inherit its tokenization scheme.\n",
- "Hugging Face makes this easy with the `AutoTokenizer`:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from transformers import AutoTokenizer\n",
- "\n",
- "tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "sample_input = dataset['train'][0]['question_title']\n",
- "print(sample_input, \"\\n\", tokenizer(sample_input))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We then map the tokenizer over our dataset:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "dataset = dataset.map(lambda x: tokenizer(x['question_title'], truncation=True), batched=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### β¨ Loading a model"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "In this case, we are loading a pre-trained network to which a custom head has been added for sequence classification.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from transformers import AutoModelForSequenceClassification\n",
- "\n",
- "model = AutoModelForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels=num_labels)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n",
- "Let's make a function to return the topic prediction from a sample question."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import torch\n",
- "\n",
- "def get_topic(sentence, tokenize=tokenizer, model=model):\n",
- " # tokenize the input\n",
- " inputs = tokenizer(sentence, return_tensors='pt')\n",
- " # ensure model and inputs are on the same device (GPU)\n",
- " inputs = {name: tensor.cuda() for name, tensor in inputs.items()}\n",
- " model = model.cuda()\n",
- " # get prediction - 10 classes \"probabilities\" (not really true because they still need to be normalized)\n",
- " with torch.no_grad():\n",
- " predictions = model(**inputs)[0].cpu().numpy()\n",
- " # get the top prediction class and convert it to its associated label\n",
- " top_prediction = predictions.argmax().item()\n",
- " return dataset['train'].features['labels'].int2str(top_prediction)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can test our prediction pipeline on a sample sentence."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "get_topic('Why is cheese so much better with wine?')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "When we ran our model, we got the answer `Politics & Government`,\n",
- "which doesn't seem quite right for a question about cheese and wine.\n",
- "\n",
- "That's because the model has not been trained yet so the outputs are still random. But at least we have a working pipeline!"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# π Log your data for better visualization\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Before we train our model, let's set up some better logging during training.\n",
- "\n",
- "Without the ability to inspect model behavior, it can be hard to debug or understand models.\n",
- "So we'll log a table of information about the model's behavior on the validation set --\n",
- "not just the loss or accuracy, but the inputs and outputs as well.\n",
- "\n",
- "Our data is already in a pandas `DataFrame`, so there's not much we have to do besides\n",
- "slightly reformat them\n",
- "and then use them to define a `ValidationDataLogger` instance."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from wandb.sdk.integration_utils.data_logging import ValidationDataLogger"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "validation_inputs = dataset['test'].remove_columns(['labels', 'attention_mask', 'input_ids'])\n",
- "validation_targets = [dataset['test'].features['labels'].int2str(x) for x in dataset['test']['labels']]\n",
- "\n",
- "validation_inputs[0], validation_targets[0]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "validation_logger = ValidationDataLogger(\n",
- " inputs = validation_inputs[:],\n",
- " targets = validation_targets\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can now log our predictions for visualization with `validation_logger.log_predictions(prediction_labels)`."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# π Training the model and logging to W&B\n",
- "\n",
- "We are now ready to fine-tune the model to solve our task.\n",
- "\n",
- "The Hugging Face [`Trainer` class](https://huggingface.co/transformers/main_classes/trainer.html)\n",
- "lets us easily train a model and is very flexible.\n",
- "\n",
- "**Note:** set `report_to` to `wandb` in `TrainingArguments` to enable logging through W&B."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from transformers import Trainer, TrainingArguments\n",
- "\n",
- "args = TrainingArguments(\n",
- " report_to='wandb', # enable logging to W&B\n",
- " output_dir='topic_classification', # set output directory\n",
- " overwrite_output_dir=True,\n",
- " evaluation_strategy='steps', # check evaluation metrics on a given # of steps\n",
- " learning_rate=5e-5, # we can customize learning rate\n",
- " max_steps=1000,\n",
- " logging_steps=100, # we will log every 100 steps\n",
- " eval_steps=500, # we will perform evaluation every 1000 steps\n",
- " eval_accumulation_steps=1, # report evaluation results after each step\n",
- " load_best_model_at_end=True,\n",
- " metric_for_best_model='accuracy',\n",
- " run_name='my_training_run' # name of the W&B run\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# automatically log model to W&B at the end\n",
- "%env WANDB_LOG_MODEL=true"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "For more customization, refer to [`TrainingArguments` documentation](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can optionally define metrics to calculate in addition to the loss through the `compute_metrics` function.\n",
- "\n",
- "Several [metrics](https://huggingface.co/metrics) are readily available from the datasets library to monitor model performance.\n",
- "\n",
- "We'll also use it to log all of our predictions at each evaluation loop,\n",
- "using the `validation_logger`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from datasets import load_metric\n",
- "import numpy as np\n",
- "\n",
- "accuracy_metric = load_metric(\"accuracy\")\n",
- "\n",
- "def compute_metrics(eval_pred):\n",
- " predictions, labels = eval_pred\n",
- " predictions = np.argmax(predictions, axis=1)\n",
- "\n",
- " # convert predictions from class (0, 1, 2β¦) to label (Health, Scienceβ¦)\n",
- " prediction_labels = [dataset['test'].features['labels'].int2str(x.item())\n",
- " for x in predictions]\n",
- " \n",
- " # log predictions\n",
- " validation_logger.log_predictions(prediction_labels)\n",
- "\n",
- " # metrics from the datasets library have a compute method\n",
- " return accuracy_metric.compute(predictions=predictions, references=labels)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The `Trainer` handles all the training & evaluation logic."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "trainer = Trainer(\n",
- " model=model, # model to be trained\n",
- " args=args, # training args\n",
- " train_dataset=dataset['train'],\n",
- " eval_dataset=dataset['test'],\n",
- " tokenizer=tokenizer, # for padding batched data\n",
- " compute_metrics=compute_metrics # for custom metrics\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "trainer.train()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can now easily observe your predictions live in your run dashboard!\n",
- "\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can monitor losses, metrics, gradients and parameters as the model trains.\n",
- "\n",
- "\n",
- "\n",
- "When training is complete, our model is logged and versioned along with its performance as metadata.\n",
- "\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "When we want to close our W&B run, we can call `wandb.finish()` (mainly useful in notebooks, called automatically in scripts)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "wandb.finish()"
- ]
- }
- ],
- "metadata": {
- "accelerator": "GPU",
- "colab": {
- "include_colab_link": true,
- "provenance": [],
- "toc_visible": true
- },
- "kernelspec": {
- "display_name": "Python 3",
- "name": "python3"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}