Skip to content

Latest commit

 

History

History
31 lines (21 loc) · 1.04 KB

README.md

File metadata and controls

31 lines (21 loc) · 1.04 KB
Raw

LeapfrogAI Whisper Backend

A LeapfrogAI API-compatible whisper wrapper for audio transcription inferencing across CPU & GPU infrastructures.

Usage

Zarf Package Deployment

To build and deploy just the whisper Zarf package (from the root of the repository):

Deploy a UDS cluster if one isn't deployed already

pip install 'ctranslate2'          # Used to download and convert the model weights
pip install 'transformers[torch]'  # Used to download and convert the model weights
make build-whisper LOCAL_VERSION=dev
uds zarf package deploy packages/whisper/zarf-package-whisper-*-dev.tar.zst --confirm

Local Development

To run the vllm backend locally without K8s (starting from the root directory of the repository):

python -m pip install src/leapfrogai_sdk
cd packages/whisper
python -m pip install ".[dev]"
ct2-transformers-converter --model openai/whisper-base --output_dir .model --copy_files tokenizer.json --quantization float32
python -u main.py