Self-Hosted Speech-to-Text: Running OpenAI Whisper on Your VPS

Transcription is everywhere now: podcast show notes, meeting recordings, support-call logs, video subtitles. The easy path is a cloud API, but those charge per minute, add up fast at volume, and mean shipping your audio (sometimes sensitive audio) to a third party. There is another option. OpenAI’s Whisper is an open speech-to-text model you can run on your own VPS, for free, with nothing leaving your server.

This article shows you how to do exactly that. We build the lightweight C++ version of Whisper on a plain CPU server, download a model, and transcribe real audio, measuring how fast it actually runs. Every command below was run before publishing.

What Whisper is, and how to run it

Whisper is a general-purpose speech recognition model from OpenAI, released openly. It is genuinely good: accurate, punctuation-aware, and multilingual out of the box. There are two common ways to run it yourself:

• openai-whisper is the official Python package. It works, but it pulls in PyTorch and a stack of dependencies that can run to several gigabytes, and it really wants a GPU to be fast.
• whisper.cpp is a port written in C and C++ with no heavy dependencies. It compiles to a single small binary, runs well on CPU, and is perfect for a VPS.

For a typical CPU-only VPS, whisper.cpp is the right tool, so that is what we will use.

Picking a model size

Whisper comes in several sizes. Bigger models are more accurate but need more RAM and run slower. Here is the practical trade-off:

Model	Size on disk	Speed on CPU	Best for
tiny	~75 MB	Fastest	Quick drafts, clear English audio
base	~142 MB	Fast	Good general default
small	~466 MB	Moderate	Noticeably better accuracy
medium	~1.5 GB	Slow on CPU	High accuracy, if you have the RAM
large	~3 GB	Slowest	Best accuracy, really wants a GPU

The English-only variants (named like tiny.en) are smaller and more accurate for English than the multilingual ones. On a modest VPS, tiny, base, or small is the sweet spot. We will use tiny.en for the demo.

Hands-on: build and transcribe

You need a Linux VPS with a build toolchain. Install it:

sudo apt-get update
sudo apt-get install -y build-essential git cmake

Build whisper.cpp

Clone the repository and compile the command-line tool. On an 8-core box this takes under a minute:

git clone --depth 1 https://github.com/ggml-org/whisper.cpp.git
cd whisper.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target whisper-cli

You now have a single binary at build/bin/whisper-cli, about 1 MB. That is the whole program.

Download a model

The repo ships a helper script. Grab the tiny English model:

bash ./models/download-ggml-model.sh tiny.en

That leaves a 75 MB file at models/ggml-tiny.en.bin. (If your box is missing wget, you can fetch it directly from Hugging Face with curl -L -o models/ggml-tiny.en.bin https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-tiny.en.bin.)

Transcribe

The repo includes a sample clip, samples/jfk.wav. Run it:

./build/bin/whisper-cli -m models/ggml-tiny.en.bin -f samples/jfk.wav

[00:00:00.000 --> 00:00:07.940]   And so my fellow Americans ask not what your country can do for you
[00:00:07.940 --> 00:00:10.760]   ask what you can do for your country.

Accurate, punctuated, and timestamped. Now the number that matters on a VPS, the speed:

main: processing 'samples/jfk.wav' (176000 samples, 11.0 sec), 4 threads ...
whisper_print_timings:    total time =   782.56 ms

It transcribed 11 seconds of audio in about 0.78 seconds on CPU. That is roughly 14 times faster than real time with the tiny model, on plain server cores with no GPU. A one-hour recording would finish in a few minutes.

Get subtitles, not just text

Whisper can write directly to subtitle and text files, which is what you usually want:

./build/bin/whisper-cli -m models/ggml-tiny.en.bin -f samples/jfk.wav -osrt -otxt -of output

That produces output.srt (subtitles with timings) and output.txt (plain transcript):

1
00:00:00,000 --> 00:00:07,940
 And so my fellow Americans ask not what your country can do for you

2
00:00:07,940 --> 00:00:10,760
 ask what you can do for your country.

Drop that .srt straight into a video player or upload it alongside a YouTube video. Whisper also supports -ovtt for WebVTT and -oj for JSON if you need structured output.

Preparing your own audio

The one requirement to remember: whisper.cpp expects 16 kHz mono WAV. Real-world audio (an MP3 podcast, an M4A voice memo) needs a quick conversion first, which ffmpeg handles:

ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav

-ar 16000 sets the sample rate, -ac 1 makes it mono. Feed the resulting WAV to whisper-cli and you are set. This one step trips up most first-timers, so build it into any script you write.

Turning it into a real service

Once the binary works, wiring it into your stack is straightforward:

• Batch a folder of recordings with a short shell loop that runs whisper-cli on each file and writes an .srt next to it.
• Expose an API by putting a thin web handler (Flask, FastAPI, or even a small Go server) in front that accepts an upload, converts it with ffmpeg, runs the binary, and returns the transcript.
• Pick threads deliberately with the -t flag. More threads help up to your core count, then stop helping.

Self-host or use an API?

Be honest about the trade-off. Self-hosting Whisper wins when:

• Privacy matters. Medical, legal, or internal audio never leaves your server.
• Volume is high. Per-minute API fees add up; your VPS cost is fixed.
• You want control. No rate limits, no surprise pricing changes, no dependency on a vendor.

An API still wins when your volume is low and spiky (a fixed VPS sitting idle is wasteful), or when you need the very top accuracy of the large model without paying for GPU hardware. For most steady, moderate workloads on a server you already run, self-hosting is the clear economic choice.

Gotchas worth knowing

• Match the model to your RAM. The model loads fully into memory. tiny and base are comfortable on a 1 to 2 GB VPS; medium and large need real headroom.
• CPU is fine for small models, painful for large ones. If you need large-level accuracy at speed, you need a GPU. Otherwise stay at small or below.
• Accuracy scales with model size and audio quality. Clean, clear speech transcribes almost perfectly even on tiny. Noisy, overlapping, or accented audio benefits from a bigger model.
• The first run downloads the model. Bake the download into your provisioning so a cold start is not a surprise.

Wrapping up

Speech-to-text used to mean a per-minute bill and handing your audio to someone else. It does not have to. With whisper.cpp, a plain CPU VPS can transcribe audio accurately, privately, and about 14 times faster than real time on a small model, all from a one-megabyte binary and a model file you download once. Build it, convert your audio to 16 kHz WAV, and you have your own transcription service with no ongoing cost and nothing leaving your server.