Skip to content

Commit

Permalink
Adding packedBERT deep dive walkthrough to Tutorials
Browse files Browse the repository at this point in the history
  • Loading branch information
arsalanu committed Apr 12, 2023
1 parent 30278aa commit 0b92ae4
Show file tree
Hide file tree
Showing 7 changed files with 1,790 additions and 0 deletions.
1 change: 1 addition & 0 deletions tutorials/blogs_code/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ This directory contains code used in Graphcore blogs

## Contents
- The [packedBERT](packedBERT) folder contains supporting code for: ["Packing: Towards 2x NLP BERT Acceleration"](https://arxiv.org/abs/2107.02027) and for the corresponding blog post: ["Introducing Packed BERT for 2x Training Speed-up in Natural Language Processing"](https://towardsdatascience.com/introducing-packed-bert-for-2x-faster-training-in-natural-language-processing-eadb749962b1).
- The [finetuning-packedBERT](finetuning-packedBERT) folder contains a supporting deep-dive walkthrough for the "Accelerating fine-tuning and inference with PackedBERT" article and the [Paperspace fine-tuning packedBERT notebooks](https://console.paperspace.com/github/gradient-ai/Graphcore-HuggingFace?file=%2Fpacked-bert).
11 changes: 11 additions & 0 deletions tutorials/blogs_code/finetuning-packedBERT/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# A deeper dive into faster fine-tuning with Packed BERT

This notebook is a walkthrough diving deeper into fine-tuning with a more online-capable version of PackedBERT, building on the original Packed BERT used for BERT pre-training. It explores all of the modifications made for preprocessing, dataset creation, the model itself and postprocessing required to implement packing for BERT (and similar models) using an example of sequence classification. The code is broken down and explained step by step, explaining the logic behind the choices made.

You can try the [notebooks](https://console.paperspace.com/github/gradient-ai/Graphcore-HuggingFace?file=%2Fpacked-bert) that use these functions in Optimum Graphcore in Paperspace.

Before starting the walkthrough, ensure you have all requirements installed with

```
pip install -r requirements.txt
```
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 5 additions & 0 deletions tutorials/blogs_code/finetuning-packedBERT/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
transformers==4.25.1
datasets
numpy
matplotlib
torch==1.13.0
1,773 changes: 1,773 additions & 0 deletions tutorials/blogs_code/finetuning-packedBERT/walkthrough.ipynb

Large diffs are not rendered by default.

0 comments on commit 0b92ae4

Please sign in to comment.