Why my transcription service ballooned to 42 GB

My local transcription service hit about 42 GB of footprint after six days of uptime, mostly idle. The cause was MLX's buffer cache: uncapped by default, never cleared, plus the Whisper and pyannote models staying resident forever.

Three fixes. Cap the cache at startup with MLX_CACHE_LIMIT_GB (default 2). After every /transcribe call, run gc.collect, mx.clear_cache and torch.mps.empty_cache. And a daemon thread unloads the whisper model and the diarization pipeline after an idle window (900 s by default, 0 disables it) — it never interrupts a running job.

/health now reports the footprint plus MLX's active, cache and peak numbers, so this doesn't need discovering by hand again.

On a test run with three back-to-back transcriptions, mlx_cache_mb read 0 after each one and the footprint held around 2.1 GB. After the idle window the models unloaded to 319 MB, then reloaded correctly on the next job.

The MPS path (diarization on GPU) wasn't exercised in this test, so that half is unverified for now.

Plain text: mlx-buffer-cache-leak.md · Atom feed · All notes