My local transcription service reached about 42 GB of memory footprint after six mostly idle days. The cause: the buffer cache of MLX, Apple's machine-learning library, uncapped by default and never cleared. And the Whisper model and pyannote's diarization pipeline, which tells speakers apart, never left memory.
Three fixes. Cap the cache at startup with MLX_CACHE_LIMIT_GB (default 2). After every /transcribe call, run gc.collect, mx.clear_cache and torch.mps.empty_cache. A background thread unloads both after an idle window (900 s by default, 0 disables it). It never interrupts a running job.
Now the /health endpoint reports the footprint and MLX's active, cache and peak numbers. No more finding this by hand.
In a test of three transcriptions in a row, mlx_cache_mb read 0 after each, and the footprint held around 2.1 GB. After the idle window the models unloaded to 319 MB, then reloaded correctly on the next job.
This test skipped the MPS path, diarization on the GPU, so that half is still unverified.
Plain text: mlx-buffer-cache-leak.md · Atom feed · All notes