MahmoudAshraf97 /whisper-diarization
Automatic Speech Recognition with Speaker Diarization based on OpenAI Whisper
★ 5,676⑂ 503👁 50⑃ 52● 272▤ 0
+1
日增长
5,676
stars
503
forks
50
watchers
52
prs
272
issues
0
releases
README
Speaker Diarization Using OpenAI Whisper
Speaker Diarization pipeline based on OpenAI Whisper
Please, star the project on github (see top-right corner) if you appreciate my contribution to the community!
What is it
This repository combines Whisper ASR capabilities with Voice Activity Detection (VAD) and Speaker Embedding to identify the speaker for each sentence in the transcription generated by Whisper. First, the vocals are extracted from the audio to increase the speaker embedding accuracy, then the transcription is generated using Whisper, then the timestamps are corrected and aligned using ctc-forced-aligner to help minimize diarization error due to time shift. The audio is then passed into MarbleNet for VAD and segmentation to exclude silences, TitaNet is then used to extract speaker embeddings to identify the speaker for each segment, the result is then associated with the timestamps generated by ctc-forced-aligner to detect the speaker for each word based on timestamps and then realigned using punctuation models to compensate for minor time shifts.
Whisper and NeMo parameters are coded into diarize.py and helpers.py, I will add the CLI arguments to change them later
Installation
Python = 3.10 is needed, 3.9 will work but you'll need to manually install the requirements one by one.
FFMPEG and Cython are needed as prerequisites to install the requirements
shell
pip install cython
or
shell
sudo apt update && sudo apt install cython3
shell
on Ubuntu or Debian
sudo apt update && sudo apt install ffmpeg
on Arch Linux
sudo pacman -S ffmpeg
on MacOS using Homebrew (https://brew.sh/)
brew install ffmpeg
on Windows using Chocolatey (https://chocolatey.org/)
choco install ffmpeg
on Windows using Scoop (https://scoop.sh/)
scoop install ffmpeg
on Windows using WinGet (https://github.com/microsoft/winget-cli)
winget install ffmpeg
shell
pip install -c constraints.txt -r requirements.txt
Usage
shell
python diarize.py -a AUDIO_FILE_NAME
If your system has enough VRAM (=10GB), you can use diarize_parallel.py instead, the difference is that it runs NeMo in parallel with Whisper, this can be beneficial in some cases and the result is the same since the two models are nondependent on each other. This is still experimental, so expect errors and sharp edges. Your feedback is welcome.
Command Line Options
- -a AUDIO_FILE_NAME: The name of the audio file to be processed
- --no-stem: Disables source separation
- --whisper-model: The model to be used for ASR, default is medium.en
- --suppress_numerals: Transcribes numbers in their pronounced letters instead of digits, improves alignment accuracy
- --device: Choose which device to use, defaults to "cuda" if available
- --language: Manually select language, useful if language detection failed
- --batch-size: Batch size for batched inference, reduce if you run out of memory, set to 0 for Top issues
#288Resulting .srt file loses synch with audio file used for transcription (Google Colab notebook)15 comments · 2025-01-22#147Diarization is not working fine for all audios14 comments · 2023-12-22#326How often do you guys have more than four speakers?11 comments · 2025-05-13#239Drop Your Feature Requests HERE11 comments · 2024-10-09#307Invalid handle. Cannot load symbol cublasLtCreate6 comments · 2025-02-25#303Right ASR Wrong Speaker diarization6 comments · 2025-02-18#309Mac M1 (ARM64)5 comments · 2025-02-27#281Diarization of larger, longer audio files does not work5 comments · 2024-12-07
Discussions
#366I made a modified Google Colab notebook with more options (line width, line count, speaker labels, etc.)0 comments · 2026-04-01#365Can't build it due to pip error0 comments · 2026-03-30#364whisper-diarization is listed on Shypd — AI Tool Directory0 comments · 2026-03-30#359MacOS Quick Action for Diarizing Audio Files0 comments · 2026-02-12#322Force Alignment1 comments · 2025-04-22#321need help1 comments · 2025-04-21#315MahmoudAshraf97 / sourman, everything OK?0 comments · 2025-03-14#287Question when running diarize.py1 comments · 2025-01-22