Fixing Painfully Slow Whisper Transcription

The silent CPU fallback trap

Related: Fix Stable Diffusion Out Of Memory on a 6GB VRAM GPU →

You ran whisper meeting_recording.mp3 --model large-v2 and stared at your terminal for an hour. The transcription might be perfectly accurate, but the process is painfully slow. If your transcription is taking almost as long as the audio runtime (a 1x real-time factor), you have a software bottleneck.

The most common reason for a crawl is that your system is silently ignoring your GPU. The default OpenAI Whisper library relies on PyTorch. If your PyTorch installation does not have the correct CUDA bindings for your specific NVIDIA driver, it falls back to your CPU.

You usually see a brief warning in your terminal before the transcription starts: UserWarning: FP16 is not supported on CPU; using FP32 instead. This is your canary in the coal mine. It means you are not using your graphics card.

To verify this, open a Python shell and check your PyTorch CUDA status:

import torch
print(torch.cuda.is_available())

If that returns False, Whisper is running on your CPU. You need to uninstall your current PyTorch package and install the CUDA-enabled version directly from the PyTorch website.

pip uninstall torch torchvision torchaudio
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

If you are on Windows using WSL2 and still see False after installing the CUDA wheel, your container runtime is likely failing to pass the hardware through. I run into this frequently, and the troubleshooting steps are nearly identical to fixing Ollama not using my NVIDIA GPU in WSL2. You need to ensure the NVIDIA Container Toolkit is installed and properly configured in your WSL environment.

Stop using the default OpenAI repository

Related: Fix ComfyUI "Torch Not Compiled With CUDA Enabled" Error →

Once your GPU is active, you might notice the speed only improves to about a 3x or 4x real-time factor. For a 60-minute file, you are still waiting 15 to 20 minutes.

The hard truth is that the openai/whisper GitHub repository is a reference implementation. OpenAI released it to prove the model works and to provide a baseline for researchers. They did not design it for high-throughput production workloads.

The default library processes audio sequentially. It takes a 30-second chunk of audio, pushes it through the neural network, generates the text, and then moves to the next 30-second chunk. It does not batch process. It does not heavily optimize the matrix multiplication. It just works.

If you want speed, you have to replace the engine. You are not changing the model itself. You are keeping the exact same Whisper weights but running them through inference engines built entirely for speed.

The drop-in replacement: faster-whisper

Related: A One-File Docker Compose Stack for Ollama and Open WebUI →

For Python developers with NVIDIA GPUs, faster-whisper is the absolute best return on investment. It is a reimplementation of Whisper using CTranslate2, which is a custom inference engine for Transformer models.

It handles quantization natively. You can load the massive large-v3 model into 8-bit precision instead of 16-bit or 32-bit. This cuts the VRAM requirement in half and dramatically accelerates the memory bandwidth, which is usually the bottleneck for Transformer models.

First, install the library:

pip install faster-whisper

Here is the exact Python script I use to transcribe large files on my RTX 3090:

from faster_whisper import WhisperModel
import time

Set compute_type="int8_float16" for massive speedups on modern GPUs

Set device="cuda" to ensure it hits the graphics card

model = WhisperModel("large-v3", device="cuda", compute_type="int8_float16")

start_time = time.time()

beam_size=5 is a good balance between speed and accuracy

segments, info = model.transcribe("meeting_recording.mp3", beam_size=5)

print(f"Detected language '{info.language}' with probability {info.language_probability}")

for segment in segments: print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

print(f"Transcription took {time.time() - start_time:.2f} seconds")

When you run this, you will see a massive leap in performance. A 60-minute file that took 15 minutes on the base repository will finish in about 3 to 4 minutes. The VRAM footprint will also hover around 4GB instead of 10GB.

The Mac and CPU savior: whisper.cpp

Related: Fix Open WebUI Showing No Models in the Dropdown →

If you are running on an Apple Silicon Mac (M1, M2, M3) or if you are deliberately running on a server without a GPU, Python and PyTorch are the wrong tools for the job. You want whisper.cpp.

Written by Georgi Gerganov in pure C/C++, this port strips out all the Python overhead and runs the Whisper model directly on the CPU or via Apple Metal. It is the exact same architectural concept that powers desktop LLM tools, so if you have ever had to troubleshoot Ollama running slow, you already know the value of optimized C++ inference.

To get it running, clone the repository and compile it. On a Mac, the make command will automatically enable Metal GPU acceleration.

git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp
make

Next, download the model weights. The repository includes a shell script for this. I recommend the quantized 4-bit or 5-bit models for Mac processors.

bash ./models/download-ggml-model.sh large-v3-q5_0

There is one strict requirement for whisper.cpp: your audio file must be a 16kHz, 16-bit WAV file. It will fail silently or throw a segmentation fault if you feed it a standard MP3. You must convert it using ffmpeg first:

ffmpeg -i meeting_recording.mp3 -ar 16000 -ac 1 -c:a pcm_s16le input_16k.wav

Then, run the inference:

./main -m models/ggml-large-v3-q5_0.bin -f input_16k.wav -t 8

The -t 8 flag sets the number of CPU threads. Do not set this higher than your physical performance cores. Setting it to your total logical core count will actually slow it down due to thread contention.

The ultimate batching tool: WhisperX

Related: Secure Ollama with Nginx, HTTPS, and a Password →

If you have a high-end GPU with 12GB to 24GB of VRAM and you need to transcribe hundreds of hours of audio, faster-whisper is good, but WhisperX is better.

WhisperX fundamentally changes how the audio is processed. Instead of feeding 30-second chunks sequentially, it runs a highly optimized Voice Activity Detection (VAD) model over the entire audio file first. The VAD model finds exactly where people are speaking and where the silence is.

It chops the audio based on those speech segments and then batches them together. It sends multiple segments to the GPU simultaneously. This pushes GPU utilization to 100% and achieves speeds up to 70x real-time.

Install it directly from the repository:

pip install git+https://github.com/m-bain/whisperx.git

You can run it straight from the command line. The --batch_size argument is the key. Start with a batch size of 16. If your GPU runs out of VRAM, drop it to 8.

whisperx meeting_recording.mp3 --model large-v3 --align_model WAV2VEC2_ASR_LARGE_LV60K_960H --batch_size 16 --compute_type int8

WhisperX also includes an alignment model. Because standard Whisper has a habit of grouping timestamps poorly, WhisperX runs a second, lighter model over the output to snap the text to exact millisecond word-level timestamps. It is the best tool available for generating subtitle files.

Real-world speed comparison

Related: How to Stop Ollama From Unloading Models (keep_alive) →

To show the actual difference these tools make, I ran a strict benchmark on my local machine.

Hardware: NVIDIA RTX 3090 (24GB VRAM), AMD Ryzen 9 5900X, 64GB RAM. Input File: A 60-minute podcast exported as a 128kbps MP3. Model: Whisper Large-v3.

WhisperX destroys the competition when you have the VRAM to support large batch sizes. The whisper.cpp result is slower here because it ran on the CPU, but it remains the undisputed champion for Mac hardware.

What did NOT work

When I first hit these bottlenecks, I wasted days trying to optimize the default repository instead of switching inference engines. Here are the dead ends I hit so you can avoid them.

Using torch.compile() PyTorch 2.0 introduced torch.compile() to optimize models at runtime. I tried wrapping the default Whisper model in it. It took 15 minutes just to compile the model graph, and then it threw a dimension mismatch error on the attention heads. It is incredibly brittle with the base repository and not worth the headache.

Manual audio chunking with ffmpeg I thought I could outsmart the sequential processing by using a bash script to slice the audio into five-minute chunks, then firing up six separate Python processes to transcribe them in parallel. This instantly maxed out my 24GB of VRAM and hard-crashed my machine. Loading the weights into memory six separate times is wildly inefficient. WhisperX handles batching at the tensor level, which is the correct way to do it.

Running base Whisper inside Docker I attempted to run the base repository inside a containerized API to serve transcriptions over my local network. The performance was identical, but I introduced entirely new networking headaches. Debugging the container routing felt exactly like the time I spent trying to fix Open WebUI in Docker cannot reach Ollama. Docker does not speed up Python execution; it only isolates it.

FAQ

Why does Whisper freeze at the end of an audio file?

Question: Why does Whisper freeze at the end of an audio file? Whisper can hallucinate on background noise or silence. It loops the same phrase trying to find the next word. Use a Voice Activity Detection (VAD) filter like WhisperX to strip silence before transcription.

Can I run the large-v3 model on 8GB of VRAM?

Question: Can I run the large-v3 model on 8GB of VRAM? Yes. Use faster-whisper and set compute_type to int8. This compresses the model weights so the large-v3 model comfortably fits inside 8GB of VRAM without a noticeable drop in accuracy.

Does audio format matter for transcription speed?

Question: Does audio format matter for transcription speed? No. Whisper resamples all input audio to 16kHz internally before processing. Compressing or changing the format of your source file will not speed up the actual neural network inference time.

Why does faster-whisper give slightly different text than OpenAI?

Question: Why does faster-whisper give slightly different text than OpenAI? Because faster-whisper uses CTranslate2, the mathematical operations are grouped differently than PyTorch. Floating point rounding differences compound over time, resulting in minor text variations.