TL;DR
Pyannote is the go-to model for speaker diarization. We’ve used it extensively while training our own Dia TTS models. But running it across millions of files took a lot of time. Actually, it was the single biggest time sink within our data pipeline. We fix this by creating a custom inference server, which is able to process 115.6 hrs of audio across 837 files in just 88.6 seconds on a single A100 GPU. This is roughly 4,700x realtime.
Diarization is a key task in speech, where the model needs to figure out who spoke when in a recording. Pyannote Community-1 is the leading open source diarization model in Diarization Error Rate (DER). We improve upon the vanilla inference code by ~50x, making it faster and cheaper for everyone to run Pyannote, while maintaining quality.
Pyannote, the King of Diarization
Pyannote’s open source models have been downloaded more than a billion times on HuggingFace. It is actually closer to a pipeline than a single model. It comprises segmentation, speaker embedding, clustering, and reconstruction of speaker timelines. We optimize inference by
- Keeping compatible work ready. We run at a concurrency of 8, and the GPU scheduler groups multiple inputs that are going through the same stage and batches work together.
- Reusing overlapping computation. By sharing frame local features between overlapping windows, we cut down on recomputation.
- Mixed precision without DER compromise. We mix TF32 and FP16 for the Segmentation model. Embedding CNN runs with INT8 convolutions and FP16 fallback. DER increases by at most 0.5 percentage points, while improving by over 1 percentage point in certain evaluation sets.
- Reducing communications between the CPU and GPU by kernel fusion, and reducing the number of kernel launches via CUDA graphing.
Results
We run the model on a variety of evaluation datasets to ensure quality does not drop.
| Dataset | Vanilla (DER) | Ours (DER) | Audio (hours) |
|---|---|---|---|
| AMI-IHM | 17.05 | 16.98 | 9.06 |
| AMI-SDM | 19.91 | 19.84 | 9.06 |
| VoxConverse | 11.14 | 11.58 | 43.54 |
| AISHELL-4 | 11.47 | 11.70 | 12.73 |
| AliMeeting | 21.15 | 19.84 | 10.78 |
| RAMC | 19.49 | 19.68 | 20.64 |
| MSDWild | 23.33 | 23.41 | 9.85 |
| Total | - | - | 115.66 |
When run sequentially with vanilla inference, this takes over an hour. With our inference server, it takes 88.6 seconds to complete all 837 files. This includes local HTTP uploads, queue waiting, audio decoding, inference, and response delivery, but excludes model loading and warmup.
Separate tests with a single request give median latency of 27 ms for 10 seconds of audio, 39 ms for 30 seconds of audio, 2.16 seconds for 1 hour and 5.93 seconds for 2 hours of audio.
Coming Soon: Pyannote Community-1 on the API
We are releasing our Pyannote Community-1 endpoints to the public in the coming weeks. We also plan to bring speaker diarization into our batched ASR offering as well. Stay tuned.
Nari Labs builds superfast inference for multimodal models. If you need custom support, talk to us and we’ll make it faster.