Perf comparison with OpenAI whisper model vs cpp model (Windows)

[vivekuppal]

Transcription using OpenAI whisper model python bindings and whisper.cpp had very similar characteristics. My expectation was that whisper.cpp would be better.

I took the binaries from Release 1.5

Transcribed the sample file jfk.wav with my OSS project Transcribe.

The results in the two cases were comparable. My initial expectation was that the results of this project would be better than the base models published by OpenAI.
I am sharing results of one run only, though I did 10+ runs with consistent results.
Also the results are consistent with other wav files besides the sample file in the repo.

whisper.cpp Results

C:\Users\vivek\Downloads\whisper-bin-x64>main.exe c:\git\whisper.cpp\samples\jfk.wav
whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-base.en.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 512
whisper_model_load: n_audio_head  = 8
whisper_model_load: n_audio_layer = 6
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 512
whisper_model_load: n_text_head   = 8
whisper_model_load: n_text_layer  = 6
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 2 (base)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: n_langs       = 99
whisper_model_load:      CPU buffer size =   156.67 MB
whisper_model_load: model size    =  156.58 MB
whisper_init_state: kv self size  =   16.52 MB
whisper_init_state: kv cross size =   18.43 MB
whisper_init_state: compute buffer (conv)   =   14.79 MB
whisper_init_state: compute buffer (encode) =   85.93 MB
whisper_init_state: compute buffer (cross)  =    4.71 MB
whisper_init_state: compute buffer (decode) =   96.41 MB

system_info: n_threads = 4 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | CUDA = 0 | COREML = 0 | OPENVINO = 0 |

main: processing 'c:\git\whisper.cpp\samples\jfk.wav' (176000 samples, 11.0 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.000 --> 00:00:11.000]   And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.


whisper_print_timings:     load time =   148.22 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     9.60 ms
whisper_print_timings:   sample time =    84.49 ms /   131 runs (    0.64 ms per run)
whisper_print_timings:   encode time =  1377.10 ms /     1 runs ( 1377.10 ms per run)
whisper_print_timings:   decode time =     9.44 ms /     2 runs (    4.72 ms per run)
whisper_print_timings:   batchd time =   245.28 ms /   125 runs (    1.96 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time =  1889.86 ms

Transcribe Results

(venv) C:\git\transcribe-main>python main.py -t C:\git\whisper.cpp\samples\jfk.wav -m base
C:\git\transcribe-main\venv\Lib\site-packages\whisper\timing.py:57: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit
[INFO] Listening to sound from Microphone: #24 - Microphone (Voicemod Virtual Audio Device (WDM))
[INFO] Listening to sound from Speaker: #20 - Headset Earphone (Corsair HS60 PRO Surround USB Sound Adapter) [Loopback]
[INFO] Adjusting for ambient noise from Default Speaker. Please play sound from Default Speaker...
[INFO] Completed ambient noise adjustment for Default Speaker.
[INFO] Whisper using GPU: True
Converting the audio file C:\git\whisper.cpp\samples\jfk.wav to text.
C:\git\whisper.cpp\samples\jfk.wav file size 352.1 kB.
Text output will be produced in transcription.txt.
Complete!
Duration(hh:mm:ss.ms) of Transcription 0:00:01.855688
[INFO] Exiting gracefully..

Output of bench for my machine is below

Operating System: Windows 10 22H2 OS Build 19045.3693
CPU: AMD Ryzen 5 3600 6-Core Processor
GPU: NVIDIA GeForce GTX 1660 SUPER
Compiler: Used binaries in Release 1.5

C:\Users\vivek\Downloads\whisper-bin-x64>bench
whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-base.en.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 512
whisper_model_load: n_audio_head  = 8
whisper_model_load: n_audio_layer = 6
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 512
whisper_model_load: n_text_head   = 8
whisper_model_load: n_text_layer  = 6
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 2 (base)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: n_langs       = 99
whisper_model_load:      CPU buffer size =   156.67 MB
whisper_model_load: model size    =  156.58 MB
whisper_init_state: kv self size  =   16.52 MB
whisper_init_state: kv cross size =   18.43 MB
whisper_init_state: compute buffer (conv)   =   14.79 MB
whisper_init_state: compute buffer (encode) =   85.93 MB
whisper_init_state: compute buffer (cross)  =    4.71 MB
whisper_init_state: compute buffer (decode) =   96.41 MB

system_info: n_threads = 4 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | CUDA = 0 | COREML = 0 | OPENVINO = 0 |

whisper_print_timings:     load time =   131.43 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     0.00 ms
whisper_print_timings:   sample time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:   encode time =  1449.71 ms /     1 runs ( 1449.71 ms per run)
whisper_print_timings:   decode time =  1012.87 ms /   256 runs (    3.96 ms per run)
whisper_print_timings:   batchd time =   576.65 ms /   320 runs (    1.80 ms per run)
whisper_print_timings:   prompt time =  6493.52 ms /  4096 runs (    1.59 ms per run)
whisper_print_timings:    total time =  9534.23 ms

[ggerganov]

You are using CPU-only whisper.cpp with 4 threads

[vivekuppal]

Thank You!

I was not using GPU binaries.
Using GPU binaries whisper.cpp consistently performs 35-40% better than the OAI Whisper model.

Whisper.cpp results

C:\Users\vivek\Downloads\whisper-cublas-bin-x64>main.exe c:\git\whisper.cpp\samples\jfk.wav
whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-base.en.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 512
whisper_model_load: n_audio_head  = 8
whisper_model_load: n_audio_layer = 6
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 512
whisper_model_load: n_text_head   = 8
whisper_model_load: n_text_layer  = 6
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 2 (base)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: n_langs       = 99
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce GTX 1660 SUPER, compute capability 7.5
whisper_backend_init: using CUDA backend
whisper_model_load:     CUDA buffer size =   156.67 MB
whisper_model_load: model size    =  156.58 MB
whisper_backend_init: using CUDA backend
whisper_init_state: kv self size  =   16.52 MB
whisper_init_state: kv cross size =   18.43 MB
whisper_init_state: compute buffer (conv)   =   14.79 MB
whisper_init_state: compute buffer (encode) =   85.93 MB
whisper_init_state: compute buffer (cross)  =    4.71 MB
whisper_init_state: compute buffer (decode) =   96.41 MB

system_info: n_threads = 4 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | CUDA = 1 | COREML = 0 | OPENVINO = 0 |

main: processing 'c:\git\whisper.cpp\samples\jfk.wav' (176000 samples, 11.0 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.000 --> 00:00:11.000]   And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.


whisper_print_timings:     load time =   334.40 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     9.21 ms
whisper_print_timings:   sample time =    66.45 ms /   131 runs (    0.51 ms per run)
whisper_print_timings:   encode time =   249.38 ms /     1 runs (  249.38 ms per run)
whisper_print_timings:   decode time =    39.69 ms /     2 runs (   19.84 ms per run)
whisper_print_timings:   batchd time =   145.81 ms /   125 runs (    1.17 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time =   856.96 ms

C:\Users\vivek\Downloads\whisper-cublas-bin-x64>

OpenAI results

(venv) C:\git\transcribe-main>python main.py -t C:\git\whisper.cpp\samples\jfk.wav -m base
C:\git\transcribe-main\venv\Lib\site-packages\whisper\timing.py:57: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit
[INFO] Listening to sound from Microphone: #24 - Microphone (Voicemod Virtual Audio Device (WDM))
[INFO] Listening to sound from Speaker: #20 - Headset Earphone (Corsair HS60 PRO Surround USB Sound Adapter) [Loopback]
[INFO] Adjusting for ambient noise from Default Speaker. Please play sound from Default Speaker...
[INFO] Completed ambient noise adjustment for Default Speaker.
[INFO] Whisper using GPU: True
[INFO] Operating in Desktop mode
Converting the audio file C:\git\whisper.cpp\samples\jfk.wav to text.
C:\git\whisper.cpp\samples\jfk.wav file size 352.1 kB.
Text output will be produced in transcription.txt.
Complete!
Duration(hh:mm:ss.ms) of Transcription 0:00:01.314196
[INFO] Exiting..

(venv) C:\git\transcribe-main>python main.py -t C:\git\whisper.cpp\samples\jfk.wav -m base
C:\git\transcribe-main\venv\Lib\site-packages\whisper\timing.py:57: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit
[INFO] Listening to sound from Microphone: #24 - Microphone (Voicemod Virtual Audio Device (WDM))
[INFO] Listening to sound from Speaker: #20 - Headset Earphone (Corsair HS60 PRO Surround USB Sound Adapter) [Loopback]
[INFO] Adjusting for ambient noise from Default Speaker. Please play sound from Default Speaker...
[INFO] Completed ambient noise adjustment for Default Speaker.
[INFO] Whisper using GPU: True
[INFO] Operating in Desktop mode
Converting the audio file C:\git\whisper.cpp\samples\jfk.wav to text.
C:\git\whisper.cpp\samples\jfk.wav file size 352.1 kB.
Text output will be produced in transcription.txt.
Complete!
Duration(hh:mm:ss.ms) of Transcription 0:00:01.318200
[INFO] Exiting..

[vivekuppal]

Any recommendations, guidance for utilizing whisper.cpp from python.
It would definitely help a lot with speeding up various functional aspects of Transcribe