Whisper transcription app big performance regression

[thundergolfer]

https://modal-com.slack.com/archives/C069RAH7X4M/p1713624663717089

[thundergolfer]

A one hour podcast used to take ~1 minute, so big drop in performance.

[thundergolfer]

I think first thing to do is to replace the use of NFS

[ahxxm]

would be great if the official example uses WhisperX, it can transcribe one hour podcast in 1 minute using only 1 container(or more specific, 1 graphi card with 16G vram, using large-v3), instead of spins up 100-300 containers for a single transcription

[ahxxm]

made a poc repo here https://github.com/ahxxm/serverless-audio-transcriber

[ahxxm]

I've been using my own dogfood for a while, this is how it looks like with A10G

The audio files I sent range from 30 minutes to 70 minutes, and I scheduled a 5 minute interval

This translates to around $0.01~$0.02 per hour of transcription, wonder how does the current official approch look like? Before and after regression, will it be cheaper and faster?

[thundergolfer]

Thanks @ahxxm this is awesome, especially the bit where you've listed benchmarks. It looks like Runpod is the cheapest? I'd bet that the
RTX A4500 $/hr rate is cheaper than our A10G.

[ahxxm]

Yeah A4500 is a new, performant(for Whisper) and cheap(Modal T4 price) one, would be great if Modal also supports that.

I think it's comparable but not a completely fair comparison, Runpod has FlashBoot that loads weights within 1 second instead of 20+ seconds, so I added a bit more CPU and memory to Modal codes

[gongouveia]

@thundergolfer @ahxxm hello, thank your for this very informative thread. I would like to ask how could I use my own model using modal, how can i send my file to the container image?

[ahxxm]

@gongouveia I doubt this is relevant to the issue, but are you asking about how to send custom model/weights into Docker image, or send the file you want to transcribe(see all examples in this repo or mine then)?