Voice Activity Detection comparison with Whisper

[egorsmkv]

Hey!

Whisper has ability to detect speech boundaries and return its timestamps.

You can use it like the following:

outputs = pipe(
    'hello.wav',
    chunk_length_s=30,
    batch_size=10,
    return_timestamps=True,
)

I would like to know how its timestamp values are in comparison with PyAnnote's timestamps.

Does someone test it already? I mean to calculate metrics on public test sets.

[hbredin]

I have not done any such comparison myself. Shouldn't be too hard to test yourself though?

That being said, whisper and pyannote target two different tasks and, in particular, they will handle short silences between words very differently. My intuition is that whisper will most likely fill short gaps between words that pyannote/segmentation-3.0 will otherwise try to keep.

Also, neither whisper nor pyannote/segmentation-3.0 have been trained specifically for VAD.