Skip to content

Latest commit

 

History

History
213 lines (143 loc) · 10.7 KB

README.md

File metadata and controls

213 lines (143 loc) · 10.7 KB
Raw

Benchmarking

Repository for AI model benchmarking on accelerator hardware.

Performance Table

Model Input Size Batch Grayskull e75 Grayskull e150 Wormhole n150 Wormhole n300 (single-chip) Wormhole n300 (dual-chip)
BERT-Large (sen/s) 384 64 83 99 118 x TBD
T5-Large (tok/s) 64 1 24 31 75 65 TBD
FLAN-T5-Large (tok/s) 64 1 9 25 70 65 TBD
Whisper-Small (tok/s) 30s 1 x x 9 9 TBD
Falcon-7B (tok/s) 128 32 x x 72 68 TBD
SD v1-4 (s/img) 512x512 1 x x 100 50 TBD
ResNet50 (img/s) 3x224x224 256 1116 1404 2893 1066 TBD
VoVNet-V2 (img/s) 3x224x224 128 516 819 1604 1199 TBD
MobileNetV1 (img/s) 3x224x224 128 2463 2912 3110 2283 TBD
MobileNetV2 (img/s) 3x224x224 256 1143 1490 2771 2433 TBD
MobileNetV3 (img/s) 3x224x224 64 1191 1267 1986 1632 TBD
HRNet-V2 (img/s) 3x224x224 128 140 231 324 258 TBD
ViT-Base (img/s) 3x224x224 64 301 363 539 446 TBD
DeiT-Base (img/s) 3x224x224 64 302 362 539 446 TBD
YOLOv5-Small (img/s) 3x320x320 128 276 221 1191 1166 TBD
OpenPose-2D (img/s) 3x224x224 64 885 1311 1260 1195 TBD
U-Net (img/s) 3x256x256 48 223 272 494 399 TBD
Inception-v4 (img/s) 3x224x224 128 371 458 1060 1118 TBD

Setup Instructions

Installation

First, create either a Python virtual environment with PyBuda installed or execute from a Docker container with PyBuda installed.

Installation instructions can be found at Install TT-BUDA.

Next, install the model requirements:

pip install -r requirements.txt

Installing on a System with a GPU

If your system contains a GPU device, you can use the requirements-cuda.txt file to install the correct dependencies.

To install packages, make sure your virtual environment is active.

To add additional CUDA packages

pip install -r requirements-cuda.txt

Environment Setup

Setup Access to HuggingFace Hub

To access the benchmarking datasets, follow these steps to set up your access to the HuggingFace Hub:

  1. Create a HuggingFace Account:

    • Visit Hugging Face and create an account if you haven't already.

  2. Generate User Access Token:

  3. Install huggingface_hub Library:

    • Install the huggingface_hub library by running:

      pip install huggingface_hub

  4. Login to HuggingFace CLI:

    • Login to the HuggingFace CLI using your User Access Token:

      huggingface-cli login

    • Enter your User Access Token when prompted.

  5. Validate Setup:

    • Run the following command to verify your login status:

      huggingface-cli whoami

    • If your username is displayed, it means you are successfully logged in.

  6. Dataset Access:

Validation Steps

After completing the setup process, ensure that everything is working correctly:

  1. Verify Hugging Face Hub Login:

    • Run the following command to verify that you are logged in to the Hugging Face Hub:

      huggingface-cli whoami

    • If your username is displayed, it means you are successfully logged in.

  2. Check Dataset Access:

  3. Accept Dataset Access (If Required):

    • If you encounter any permission errors while accessing the ImageNet-1k dataset, ensure that you follow the instructions provided on the dataset page to grant access.

Benchmarking Datasets Setup

To set up the three required datasets for running benchmarking tests within this repository, follow these steps for each dataset:

  1. HuggingFace datasets: will download into ${HF_HOME}/datasets/ once HuggingFace Hub access is set up.

  2. COCO Dataset: You can automatically download the COCO dataset from here. No login is required, and the dataset will be cached in ~/.cache/coco.

    To download the COCO dataset, follow these steps:

    # use another location for MLDATA_DIR if desired, below is default
# Create the `coco` directory inside the cache directory:
mkdir -p ~/.cache/mldata/coco

# Navigate to the `coco` directory:
cd ~/.cache/mldata/coco

# Create the `images` directory:
mkdir images
cd images

# Download the COCO validation images:
wget http://images.cocodataset.org/zips/val2017.zip

# Unzip the downloaded file:
unzip val2017.zip

# Move back to the `coco` directory:
cd ..

# Download the COCO train/val annotations:
wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip

# Unzip the downloaded file:
unzip annotations_trainval2017.zip

  3. LGG Segmentation Dataset: must be manually downloaded from https://www.kaggle.com/datasets/mateuszbuda/lgg-mri-segmentation, this requires a Kaggle login. The commands below are to extract the downloaded archive into the correct location: ~/.cache/mldata/lgg_segmentation/kaggle_3m.

    # use another location for MLDATA_DIR if desired, below is default;
# Download and move the downloaded archive and unzip within the llgg_segmentation folder.
mkdir -p ~/.cache/mldata/lgg_segmentation
cd ~/.cache/mldata/lgg_segmentation
# download and move the archive here then unzip
unzip archive.zip
# the dataset appears to have two copies that are equivalent, remove the extra one
rm -r lgg-mri-segmentation

Run Benchmarking

As part of the environment setup, you may need to add the root to PYTHONPATH:

export PYTHONPATH=.

Benchmarking Script

benchmark.py allows easy way to benchmark performance of a support model, while varying configurations and compiler options. The script will measure real end-to-end time on host, starting post-compile at first input pushed, and ending at last output received.

To specify the device to run on ("tt", "cpu", or "cuda"), include the -d argument flag.

The script optionally outputs a .json file with benchmark results and options used to run the test. If the file already exists, the results will be appended, allowing the user to run multiple benchmark back-to-back, like:

python benchmark.py -d tt -m bert -c base --task text_classification --save_output

or

python benchmark.py -d cuda -m bert -c base --task text_classification --save_output

To see which models and configurations are currently supported, run:

benchmark.py --list

Run Examples

You can find example commands for various conditions in the file:

  • run_benchmark_tt_perf for TT and run_benchmark_cuda GPU & CPU devices

Contributing

We are excited to move our development to the public, open-source domain. However, we are not adequately staffed to review contributions in an expedient and manageable time frame at this time. In the meantime, please review the contributor's guide for more information about contribution standards.

Communication

If you would like to formally propose a new feature, report a bug, or have issues with permissions, please file through GitHub issues.

Please access the Discord community forum for updates, tips, live troubleshooting support, and more!