v1.7.5 #2995

ggerganov · 2025-04-02T14:34:09Z

ggerganov
Apr 2, 2025
Maintainer

Overview

This is a relatively big update with various build and CI improvements especially for iOS and WASM. There are also some performance gains, especially for the Metal backend and probably for Arm-based devices.

Big shoutout to @danbev for stepping up and completing the maintenance roadmap for this release!

Mobile examples

All mobile examples have been refreshed. The iOS examples specifically are now much easier to build thanks to the new XCFramework workflow. This should simplify significantly integration of whisper.cpp in 3rd party iOS and macOS apps. CoreML build and convert instructions have also been updated.

WASM examples

The WASM examples are now automatically updated on each new commit and hosted in Github Pages at https://ggerganov.github.io/whisper.cpp/. Problems with CORS rules should be resolved.

Some performance numbers for this release:

M2 Ultra

Flash Attention ON:

CPU	Config	Model	Th	FA	Enc.	Dec.	Bch5	PP	Commit
M2 ULTRA	METAL	tiny	1	1	7.82	1.31	0.35	0.01	`ad4e350`
M2 ULTRA	METAL	tiny-q5_0	1	1	8.32	1.28	0.37	0.01	`ad4e350`
M2 ULTRA	METAL	tiny-q5_1	1	1	8.21	1.28	0.37	0.01	`ad4e350`
M2 ULTRA	METAL	tiny-q8_0	1	1	7.97	1.23	0.36	0.01	`ad4e350`
M2 ULTRA	METAL	base	1	1	13.96	1.80	0.42	0.02	`ad4e350`
M2 ULTRA	METAL	base-q5_0	1	1	15.19	1.75	0.42	0.02	`ad4e350`
M2 ULTRA	METAL	base-q5_1	1	1	15.09	1.75	0.42	0.02	`ad4e350`
M2 ULTRA	METAL	base-q8_0	1	1	14.45	1.70	0.41	0.02	`ad4e350`
M2 ULTRA	METAL	small	1	1	40.08	3.54	0.86	0.05	`ad4e350`
M2 ULTRA	METAL	small-q5_0	1	1	45.07	3.51	0.88	0.05	`ad4e350`
M2 ULTRA	METAL	small-q5_1	1	1	45.05	3.52	0.88	0.05	`ad4e350`
M2 ULTRA	METAL	small-q8_0	1	1	42.04	3.34	0.85	0.05	`ad4e350`
M2 ULTRA	METAL	medium	1	1	107.20	7.28	1.79	0.11	`ad4e350`
M2 ULTRA	METAL	medium-q5_0	1	1	125.02	6.67	1.83	0.12	`ad4e350`
M2 ULTRA	METAL	medium-q5_1	1	1	124.83	6.70	1.84	0.12	`ad4e350`
M2 ULTRA	METAL	medium-q8_0	1	1	114.56	6.53	1.79	0.11	`ad4e350`
M2 ULTRA	METAL	medium-dis	1	1	95.96	1.01	0.23	0.01	`ad4e350`
M2 ULTRA	METAL	large-v2	1	1	194.29	10.57	2.67	0.20	`ad4e350`
M2 ULTRA	METAL	large-v2-q5_0	1	1	230.74	9.57	2.73	0.23	`ad4e350`
M2 ULTRA	METAL	large-v2-q5_1	1	1	229.97	9.69	2.74	0.23	`ad4e350`
M2 ULTRA	METAL	large-v2-q8_0	1	1	208.11	9.37	2.60	0.21	`ad4e350`
M2 ULTRA	METAL	large-v2-dis	1	1	172.72	1.12	0.26	0.02	`ad4e350`
M2 ULTRA	METAL	large-v3-turbo	1	1	174.46	1.74	0.42	0.03	`ad4e350`
M2 ULTRA	METAL	large-v3-turbo-q5_0	1	1	205.78	1.54	0.42	0.04	`ad4e350`
M2 ULTRA	METAL	large-v3-turbo-q8_0	1	1	186.33	1.50	0.40	0.03	`ad4e350`

Flash Attention OFF:

CPU	Config	Model	Th	Enc.	Dec.	Bch5	PP	Commit
M2 ULTRA	METAL	tiny	1	8.74	1.20	0.36	0.01	`ad4e350`
M2 ULTRA	METAL	tiny-q5_0	1	10.30	1.15	0.38	0.01	`ad4e350`
M2 ULTRA	METAL	tiny-q5_1	1	10.71	1.13	0.38	0.01	`ad4e350`
M2 ULTRA	METAL	tiny-q8_0	1	9.97	1.12	0.37	0.01	`ad4e350`
M2 ULTRA	METAL	base	1	16.77	1.71	0.44	0.02	`ad4e350`
M2 ULTRA	METAL	base-q5_0	1	16.92	1.63	0.44	0.02	`ad4e350`
M2 ULTRA	METAL	base-q5_1	1	16.84	1.63	0.44	0.02	`ad4e350`
M2 ULTRA	METAL	base-q8_0	1	16.12	1.63	0.44	0.02	`ad4e350`
M2 ULTRA	METAL	small	1	45.29	3.44	0.92	0.05	`ad4e350`
M2 ULTRA	METAL	small-q5_0	1	50.43	3.34	0.94	0.06	`ad4e350`
M2 ULTRA	METAL	small-q5_1	1	50.49	3.35	0.93	0.06	`ad4e350`
M2 ULTRA	METAL	small-q8_0	1	47.37	3.20	0.91	0.05	`ad4e350`
M2 ULTRA	METAL	medium	1	122.81	7.39	1.99	0.12	`ad4e350`
M2 ULTRA	METAL	medium-q5_0	1	140.62	6.73	2.03	0.14	`ad4e350`
M2 ULTRA	METAL	medium-q5_1	1	140.44	6.74	2.04	0.14	`ad4e350`
M2 ULTRA	METAL	medium-q8_0	1	131.05	6.54	1.95	0.13	`ad4e350`
M2 ULTRA	METAL	medium-dis	1	110.95	0.99	0.24	0.02	`ad4e350`
M2 ULTRA	METAL	large-v2	1	222.19	10.93	3.01	0.21	`ad4e350`
M2 ULTRA	METAL	large-v2-q5_0	1	258.47	9.75	3.01	0.25	`ad4e350`
M2 ULTRA	METAL	large-v2-q5_1	1	258.40	9.85	3.01	0.24	`ad4e350`
M2 ULTRA	METAL	large-v2-q8_0	1	236.68	9.61	2.85	0.23	`ad4e350`
M2 ULTRA	METAL	large-v2-dis	1	199.28	1.12	0.27	0.02	`ad4e350`
M2 ULTRA	METAL	large-v3-turbo	1	201.49	1.76	0.45	0.03	`ad4e350`
M2 ULTRA	METAL	large-v3-turbo-q5_0	1	233.70	1.55	0.46	0.04	`ad4e350`
M2 ULTRA	METAL	large-v3-turbo-q8_0	1	214.20	1.51	0.44	0.04	`ad4e350`

M4 Max

Flash Attention ON:

CPU	Config	Model	Th	FA	Enc.	Dec.	Bch5	PP	Commit
M4 Max	METAL	tiny	1	1	15.22	0.89	0.26	0.01	`ad4e350`
M4 Max	METAL	tiny-q8_0	1	1	14.70	0.86	0.26	0.01	`ad4e350`
M4 Max	METAL	base	1	1	25.33	1.36	0.30	0.02	`ad4e350`
M4 Max	METAL	base-q8_0	1	1	21.27	1.31	0.30	0.02	`ad4e350`
M4 Max	METAL	small	1	1	58.43	2.78	0.60	0.05	`ad4e350`
M4 Max	METAL	small-q8_0	1	1	60.26	2.39	0.60	0.05	`ad4e350`
M4 Max	METAL	medium	1	1	169.73	6.03	1.31	0.14	`ad4e350`
M4 Max	METAL	medium-q8_0	1	1	176.61	4.99	1.31	0.14	`ad4e350`
M4 Max	METAL	large-v2	1	1	316.18	9.60	2.08	0.24	`ad4e350`
M4 Max	METAL	large-v2-q8_0	1	1	329.59	7.55	2.08	0.25	`ad4e350`

Flash Attention OFF:

CPU	Config	Model	Th	Enc.	Dec.	Bch5	PP	Commit
M4 Max	METAL	tiny	1	13.12	0.87	0.29	0.01	`ad4e350`
M4 Max	METAL	tiny-q8_0	1	15.90	0.88	0.31	0.01	`ad4e350`
M4 Max	METAL	base	1	23.10	1.42	0.34	0.02	`ad4e350`
M4 Max	METAL	base-q8_0	1	27.25	1.31	0.34	0.02	`ad4e350`
M4 Max	METAL	small	1	71.76	3.02	0.70	0.06	`ad4e350`
M4 Max	METAL	small-q8_0	1	73.88	2.60	0.71	0.06	`ad4e350`
M4 Max	METAL	medium	1	208.22	6.94	1.55	0.16	`ad4e350`
M4 Max	METAL	medium-q8_0	1	214.65	5.90	1.57	0.17	`ad4e350`
M4 Max	METAL	large-v2	1	381.72	11.28	2.51	0.29	`ad4e350`
M4 Max	METAL	large-v2-q8_0	1	394.97	8.90	2.45	0.30	`ad4e350`

V100

Flash Attention ON:

GPU	Config	Model	Th	FA	Enc.	Dec.	Bch5	PP	Commit
V100	AVX2 CUDA	tiny	8	1	4.01	0.90	0.25	0.01	`ad4e350`
V100	AVX2 CUDA	tiny-q5_1	8	1	4.12	0.88	0.18	0.01	`ad4e350`
V100	AVX2 CUDA	base	8	1	7.00	1.30	0.35	0.01	`ad4e350`
V100	AVX2 CUDA	base-q5_1	8	1	7.22	1.21	0.26	0.02	`ad4e350`
V100	AVX2 CUDA	small	8	1	18.68	2.39	0.69	0.03	`ad4e350`
V100	AVX2 CUDA	small-q5_1	8	1	19.38	2.32	0.51	0.03	`ad4e350`
V100	AVX2 CUDA	medium	8	1	53.17	5.15	1.45	0.06	`ad4e350`
V100	AVX2 CUDA	medium-q5_0	8	1	55.09	4.64	1.05	0.07	`ad4e350`
V100	AVX2 CUDA	large-v2	8	1	85.77	7.57	2.19	0.10	`ad4e350`
V100	AVX2 CUDA	large-v2-q5_0	8	1	89.24	6.48	1.48	0.11	`ad4e350`
V100	AVX2 CUDA	large-v3-turbo	8	1	75.56	1.25	0.37	0.02	`ad4e350`
V100	AVX2 CUDA	large-v3-turbo-q5_0	8	1	78.48	1.01	0.24	0.02	`ad4e350`

Flash Attention OFF:

GPU	Config	Model	Th	Enc.	Dec.	Bch5	PP	Commit
V100	AVX2 CUDA	tiny	8	6.15	1.02	0.30	0.01	`ad4e350`
V100	AVX2 CUDA	tiny-q5_1	8	5.92	0.96	0.25	0.01	`ad4e350`
V100	AVX2 CUDA	base	8	10.60	1.43	0.43	0.02	`ad4e350`
V100	AVX2 CUDA	base-q5_1	8	10.80	1.37	0.36	0.02	`ad4e350`
V100	AVX2 CUDA	small	8	31.83	2.82	0.87	0.04	`ad4e350`
V100	AVX2 CUDA	small-q5_1	8	31.88	2.68	0.72	0.04	`ad4e350`
V100	AVX2 CUDA	medium	8	81.30	6.02	1.81	0.09	`ad4e350`
V100	AVX2 CUDA	medium-q5_0	8	83.21	5.44	1.41	0.10	`ad4e350`
V100	AVX2 CUDA	large-v2	8	134.81	8.64	2.69	0.14	`ad4e350`
V100	AVX2 CUDA	large-v2-q5_0	8	138.95	7.57	2.04	0.15	`ad4e350`
V100	AVX2 CUDA	large-v3-turbo	8	124.42	1.37	0.43	0.02	`ad4e350`
V100	AVX2 CUDA	large-v3-turbo-q5_0	8	127.81	1.13	0.32	0.03	`ad4e350`

What's Changed

docs: Fix main -> whisper-cli in download scripts by @domdomegg in docs: Fix main -> whisper-cli in download scripts #2707
Adding back docker instructions in v1.7.4 by @jayant-yadav in Adding back docker instructions in v1.7.4 #2711
Expose whisper_full_get_segment_no_speech_prob_from_state by @sandrohanea in Expose whisper_full_get_segment_no_speech_prob_from_state #2716
Use Unique Filenames for FFmpeg Conversion to Prevent File Overwrites by @NETZkultur in Use Unique Filenames for FFmpeg Conversion to Prevent File Overwrites #2718
talk-llama : sync llama.cpp by @ggerganov in talk-llama : sync llama.cpp #2709
whisper : fix gpu device selection by @ggerganov in whisper : fix gpu device selection #2728
sync : ggml by @ggerganov in sync : ggml #2737
Fix whisper.objc Xcode project by @iamcgn in Fix whisper.objc Xcode project #2736
ruby : Make context accept initial parameters, API to retrieve a segment and more by @KitaitiMakoto in ruby : Make context accept initial parameters, API to retrieve a segment and more #2749
sync : ggml by @ggerganov in sync : ggml #2779
Fix coreml export script by @mgrachten in Fix coreml export script #2770
Add max_len param in node addon by @billyct in Add max_len param in node addon #2760
cmake : fix compile assumptions for power9/etc by @midnightmagic in cmake : fix compile assumptions for power9/etc #2777
Fixes for Windows by @foldl in Fixes for Windows #2790
Restore big endian support by @fitzsim in Restore big endian support #2816
Add beam size parameter to stream example by @masahji in Add beam size parameter to stream example #2836
sync : ggml by @ggerganov in sync : ggml #2844
Adjusted stream example to stop on ^C when no audio is received. by @petterreinholdtsen in Adjusted stream example to stop on ^C when no audio is received. #2822
Use miniaudio for direct decoding flac, mp3, ogg and wav by @data-man in Use miniaudio for direct decoding flac, mp3, ogg and wav #2759
common : try to fix build by @ggerganov in common : try to fix build #2845
common : separate whisper sources by @ggerganov in common : separate whisper sources #2846
whisper : support GGML_BACKEND_DL by @slaren in whisper : support GGML_BACKEND_DL #2843
ruby : follow audio library change by @KitaitiMakoto in ruby : follow audio library change #2851
fix: missing include common-whisper in addon.node by @buxuku in fix: missing include common-whisper in addon.node #2858
Fixes audio loading by miniaudio by @data-man in Fixes audio loading by miniaudio #2862
Updated models download URL by @AMDphreak in Updated models download URL #2756
[Fix] When m_audio_pos overflows, m_audio_len should also increase in… by @Ivy233 in [Fix] When m_audio_pos overflows, m_audio_len should also increase in… #2855
sync : ggml by @ggerganov in sync : ggml #2868
whisper: add xcframework build script by @mdestagnol in whisper: add xcframework build script #2873
examples : add dl to the list of libraries linked by @danbev in examples : add dl to the list of libraries linked #2875
ggml-ci: add run.sh by @redraskal in ggml-ci: add run.sh #2877
ggml-ci: update input env variables to GG_BUILD_ by @redraskal in ggml-ci: update input env variables to GG_BUILD_ #2879
examples : add GGML_USE_CPU=ON flag to whisper.objc by @danbev in examples : add GGML_USE_CPU=ON flag to whisper.objc #2880
Update convert-h5-to-ggml.py by @fltman in Update convert-h5-to-ggml.py #2840
whisper : add option to use system-installed GGML by @peter277 in whisper : add option to use system-installed GGML #2887
examples : use xcframework in whisper.objc example by @danbev in examples : use xcframework in whisper.objc example #2882
ci : add release job and include xcframework by @danbev in ci : add release job and include xcframework #2889
whisper : enable compiler warnings for src by @danbev in whisper : enable compiler warnings for src #2891
ci : add missing env.branch_name to build.yml by @danbev in ci : add missing env.branch_name to build.yml #2896
whisper : fix compiler warnings in whisper.cpp by @danbev in whisper : fix compiler warnings in whisper.cpp #2895
ci : add ccache action to windows-cublas job by @danbev in ci : add ccache action to windows-cublas job #2893
Implementing Encoder Begin Callback for golang binding by @aderbedr in Implementing Encoder Begin Callback for golang binding #2900
ci : refactor cuda toolkit installation steps by @danbev in ci : refactor cuda toolkit installation steps #2902
examples : command.wasm updates by @danbev in examples : command.wasm updates #2904
examples : update wasm examples to include server.py [no ci] by @danbev in examples : update wasm examples to include server.py [no ci] #2908
ci : use ninja and fix caching for windows-cublas by @danbev in ci : use ninja and fix caching for windows-cublas #2910
examples : add WHISPER_SDL2 check to deprecation executables by @danbev in examples : add WHISPER_SDL2 check to deprecation executables #2911
xcframework : add support for CoreML to ios/macOS by @danbev in xcframework : add support for CoreML to ios/macOS #2912
ci : increase windows-cublas evict-old-files to 5d by @danbev in ci : increase windows-cublas evict-old-files to 5d #2915
examples : update whisper.objc README.md by @danbev in examples : update whisper.objc README.md #2916
whisper : add check for CPU backend initialization by @danbev in whisper : add check for CPU backend initialization #2918
readme : update Python version to 3.11 for Core ML support [no -ci] by @danbev in readme : update Python version to 3.11 for Core ML support [no -ci] #2919
whisper.swiftui : Add Core ML support to README [no ci] by @danbev in whisper.swiftui : Add Core ML support to README [no ci] #2921
whisper : update default model download directory behavior to use current working directory when script is in /bin/ directory by @peter277 in whisper : update default model download directory behavior to use current working directory when script is in /bin/ directory #2924
ci : remove CMAKE_CUDA_ARCHITECTURES in windows-cublas by @danbev in ci : remove CMAKE_CUDA_ARCHITECTURES in windows-cublas #2923
whisper : initialize decoder's rng with unique seed by @danbev in whisper : initialize decoder's rng with unique seed #2932
whisper : enhance model download scripts functionality and resolve compiler warning by @peter277 in whisper : enhance model download scripts functionality and resolve compiler warning #2925
ggml : add logging for native build options/vars by @danbev in ggml : add logging for native build options/vars #2935
examples : fix request path for local worker files by @danbev in examples : fix request path for local worker files #2937
examples : fix nthread parsing in whisper.wasm by @danbev in examples : fix nthread parsing in whisper.wasm #2938
examples : reduce initial memory to 512MB by @danbev in examples : reduce initial memory to 512MB #2939
ci: fix SYCL build by @qnixsynapse in ci: fix SYCL build #2943
whisper.android.java : update build with ggml source changes by @danbev in whisper.android.java : update build with ggml source changes #2942
whisper.android : add GGML_USE_CPU compile definition by @danbev in whisper.android : add GGML_USE_CPU compile definition #2945
Update README.md by @Page-MS in Update README.md #2946
bindings.javascript : update test instructions [no ci] by @danbev in bindings.javascript : update test instructions [no ci] #2951
bindings.java : enable copyLibs task [no ci] by @danbev in bindings.java : enable copyLibs task [no ci] #2949
whisper : add support for backends with multiple ggml_backend_buffer_type by @eddnjjn in whisper : add support for backends with multiple ggml_backend_buffer_type #2863
bindings-go : update Makefile to use cmake by @danbev in bindings-go : update Makefile to use cmake #2952
sync : ggml by @ggerganov in sync : ggml #2953
support progress_callback API for addon.node by @buxuku in support progress_callback API for addon.node #2941
bindings.ruby : fix test failures in test_whisper by @danbev in bindings.ruby : fix test failures in test_whisper #2955
Adding in DetectedLanguage to go bindings by @aderbedr in Adding in DetectedLanguage to go bindings #2947
sync : ggml by @ggerganov in sync : ggml #2962
whisper : remove unnecessary GGML_UNUSED macro by @danbev in whisper : remove unnecessary GGML_UNUSED macro #2960
feat: add health check endpoint to server by @sachaarbonel in feat: add health check endpoint to server #2968
ci : add github pages workflow for wasm examples by @danbev in ci : add github pages workflow for wasm examples #2969
examples : update README links to point to pages deployment by @danbev in examples : update README links to point to pages deployment #2971
cmake: improve Vulkan cooperative matrix support checks by @sandrohanea in cmake: improve Vulkan cooperative matrix support checks #2966
sync : ggml by @ggerganov in sync : ggml #2972
ci : re-enable android_java job by @danbev in ci : re-enable android_java job #2958
ci : re-enable freeBDS-latest job by @danbev in ci : re-enable freeBDS-latest job #2973
android.java : re-add ggml source updates by @danbev in android.java : re-add ggml source updates #2975
tests : re-enable tests [no ci] by @danbev in tests : re-enable tests [no ci] #2977
ci : add coreml job that converts base.en to coreml [no ci] by @danbev in ci : add coreml job that converts base.en to coreml [no ci] #2981
coreml: fix Whisper to CoreML conversion by disabling SDPA [no ci] by @danbev in coreml: fix Whisper to CoreML conversion by disabling SDPA [no ci] #2979
whisper.objc : fix typo in README.md [no ci] by @danbev in whisper.objc : fix typo in README.md [no ci] #2985
ci : remove intermediate build on push to master by @danbev in ci : remove intermediate build on push to master #2986
examples : clarify Core ML encoder model usage [no ci] by @danbev in examples : clarify Core ML encoder model usage [no ci] #2987
tests : remove gh label test-whisper-cli-tiny-en by @danbev in tests : remove gh label test-whisper-cli-tiny-en #2988
sync : ggml by @ggerganov in sync : ggml #2991
bench : update numbers [no ci] by @ggerganov in bench : update numbers [no ci] #2993
release : v1.7.5 by @ggerganov in release : v1.7.5 #2994

New Contributors

@domdomegg made their first contribution in docs: Fix main -> whisper-cli in download scripts #2707
@jayant-yadav made their first contribution in Adding back docker instructions in v1.7.4 #2711
@NETZkultur made their first contribution in Use Unique Filenames for FFmpeg Conversion to Prevent File Overwrites #2718
@iamcgn made their first contribution in Fix whisper.objc Xcode project #2736
@mgrachten made their first contribution in Fix coreml export script #2770
@billyct made their first contribution in Add max_len param in node addon #2760
@midnightmagic made their first contribution in cmake : fix compile assumptions for power9/etc #2777
@foldl made their first contribution in Fixes for Windows #2790
@masahji made their first contribution in Add beam size parameter to stream example #2836
@data-man made their first contribution in Use miniaudio for direct decoding flac, mp3, ogg and wav #2759
@buxuku made their first contribution in fix: missing include common-whisper in addon.node #2858
@AMDphreak made their first contribution in Updated models download URL #2756
@Ivy233 made their first contribution in [Fix] When m_audio_pos overflows, m_audio_len should also increase in… #2855
@mdestagnol made their first contribution in whisper: add xcframework build script #2873
@danbev made their first contribution in examples : add dl to the list of libraries linked #2875
@redraskal made their first contribution in ggml-ci: add run.sh #2877
@fltman made their first contribution in Update convert-h5-to-ggml.py #2840
@peter277 made their first contribution in whisper : add option to use system-installed GGML #2887
@aderbedr made their first contribution in Implementing Encoder Begin Callback for golang binding #2900
@qnixsynapse made their first contribution in ci: fix SYCL build #2943
@Page-MS made their first contribution in Update README.md #2946
@eddnjjn made their first contribution in whisper : add support for backends with multiple ggml_backend_buffer_type #2863

Full Changelog: v1.7.4...v1.7.5

This discussion was created from the release v1.7.5.

ggerganov · 2025-04-02T14:36:29Z

ggerganov
Apr 2, 2025
Maintainer Author

@danbev We should generate and attach the XCFramework using the latest commit and attach it to the release assets. Do you have access to do that: https://github.com/ggerganov/whisper.cpp/releases/edit/v1.7.5

4 replies

danbev Apr 2, 2025
Maintainer

We should generate and attach the XCFramework using the latest commit and attach it to the release assets.

Yep, I have access to that and I'll build it an attach it 👍

danbev Apr 2, 2025
Maintainer

@ggerganov Should be visible there now.

ggerganov Apr 2, 2025
Maintainer Author

Btw, in the past we used to generate Windows binaries and attach them to the release assets (for example see https://github.com/ggerganov/whisper.cpp/releases/tag/v1.5.2). But I don't remember why at some point I disabled those - think there was some issue with the CI. We should probably take a look at re-enabling those at some point.

danbev Apr 2, 2025
Maintainer

I was actually expecting the xcframework to be attached automatically by the CI release build, so I'll look into that and take a look at the Windows binaries as well.

JulianPscheid · 2025-04-03T18:06:13Z

JulianPscheid
Apr 3, 2025

Fantastic work @ggerganov @danbev! The xcframework makes building on apple devices so much easier (wish I had had that 6 months ago... would have saved me lots of headaches 😅). And the CPU processing appears to be significantly faster compared to 1.7.4.

3 replies

ggerganov Apr 3, 2025
Maintainer Author

Thanks for the feedback - happy to hear!

Regarding the CPU performance, I believe there should be a significant boost for Arm-based CPUs using quantized models and the KleidiAI inference kernels. This was added in #2863, but I haven't had a chance to give this a try and also there aren't build instructions yet. But if anyone is interested, they can reach-out for support in the discussions/issues. (cc @eddnjjn in case you have some additional insights)

eddnjjn Apr 4, 2025

#2863 adds support for tensor_traits (see ggml/src/ggml-cpu/ggml-cpu-traits.h) which are used to accelerate the CPU backend by offloading specific operators onto optimized operator implementations. Currently there are 3 implementations: aarch64, amx and Arm® KleidiAI™.

When building for Arm, aarch64 is enabled by default and this gives a nice boost for Q4_0 matmul operations. KleidiAI replaces aarch64 if you build with -DGGML_CPU_KLEIDIAI=ON.

The current KleidiAI integration accelerates Q4_0 matmul operations through the use of different features of the Arm ISA (dotprod, int8mm and SME). On devices that support SME, you enable the SME kernels by setting the environment variable GGML_KLEIDIAI_SME=1.

Sonnenfleck Apr 4, 2025

Core ML also got a nice speed boost. I'm getting transcription speeds above 40:1 with 1.7.5 (Mac Mini M4 Pro, large-v3-turbo, language German). With 1.7.4, I remember it being closer to 30:1. Great optimization work!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v1.7.5 #2995

{{title}}

Replies: 2 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

v1.7.5 #2995

ggerganov Apr 2, 2025 Maintainer

Overview

Mobile examples

WASM examples

M2 Ultra

M4 Max

V100

What's Changed

New Contributors

Replies: 2 comments · 7 replies

ggerganov Apr 2, 2025 Maintainer Author

danbev Apr 2, 2025 Maintainer

danbev Apr 2, 2025 Maintainer

ggerganov Apr 2, 2025 Maintainer Author

danbev Apr 2, 2025 Maintainer

JulianPscheid Apr 3, 2025

ggerganov Apr 3, 2025 Maintainer Author

eddnjjn Apr 4, 2025

Sonnenfleck Apr 4, 2025

ggerganov
Apr 2, 2025
Maintainer

Replies: 2 comments 7 replies

ggerganov
Apr 2, 2025
Maintainer Author

danbev Apr 2, 2025
Maintainer

danbev Apr 2, 2025
Maintainer

ggerganov Apr 2, 2025
Maintainer Author

danbev Apr 2, 2025
Maintainer

JulianPscheid
Apr 3, 2025

ggerganov Apr 3, 2025
Maintainer Author