Embeddings and GPU usage

[MadsRC]

When generating embeddings (both using the embedding utility compiled from this repo, and also using langchain and llama-cpp-python) on a MacBook M1, I've noticed that there seem to be no additional load on the GPU, but rather puts the load on the CPU.

It's most likely me that is missing something, but I was under the impression that generating the embeddings could be GPU accelerated as well?

When prompting the model with the main utility (and also langchain and llama-cpp-python), the GPU sees usage. Looking at the output, it does seem like Metal is being initiated:

llama.cpp: loading model from /Users/user/projects/models/llama2/llama-2-7b-chat/ggml-model-q4_0.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_head_kv  = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: n_gqa      = 1
llama_model_load_internal: rnorm_eps  = 1.0e-06
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =    0.08 MB
llama_model_load_internal: mem required  = 3949.96 MB (+  256.00 MB per state)
llama_new_context_with_model: kv self size  =  256.00 MB
ggml_metal_init: allocating
ggml_metal_init: using MPS
ggml_metal_init: loading '/Users/user/projects/llmExperiments/venv/lib/python3.11/site-packages/llama_cpp/ggml-metal.metal'
ggml_metal_init: loaded kernel_add                            0x127c0a610
ggml_metal_init: loaded kernel_add_row                        0x127c0a870
ggml_metal_init: loaded kernel_mul                            0x127c0aad0
ggml_metal_init: loaded kernel_mul_row                        0x127c0ad30
ggml_metal_init: loaded kernel_scale                          0x127c0af90
ggml_metal_init: loaded kernel_silu                           0x127c0b1f0
ggml_metal_init: loaded kernel_relu                           0x127c0b450
ggml_metal_init: loaded kernel_gelu                           0x127c0b6b0
ggml_metal_init: loaded kernel_soft_max                       0x127c0b910
ggml_metal_init: loaded kernel_diag_mask_inf                  0x127c0bb70
ggml_metal_init: loaded kernel_get_rows_f16                   0x127c0bdd0
ggml_metal_init: loaded kernel_get_rows_q4_0                  0x127c0c030
ggml_metal_init: loaded kernel_get_rows_q4_1                  0x127c0c290
ggml_metal_init: loaded kernel_get_rows_q2_K                  0x127c0c4f0
ggml_metal_init: loaded kernel_get_rows_q3_K                  0x127c0c750
ggml_metal_init: loaded kernel_get_rows_q4_K                  0x127c0c9b0
ggml_metal_init: loaded kernel_get_rows_q5_K                  0x127c0cc10
ggml_metal_init: loaded kernel_get_rows_q6_K                  0x127c0ce70
ggml_metal_init: loaded kernel_rms_norm                       0x127c0d0d0
ggml_metal_init: loaded kernel_norm                           0x127c0d5c0
ggml_metal_init: loaded kernel_mul_mat_f16_f32                0x127c0db60
ggml_metal_init: loaded kernel_mul_mat_q4_0_f32               0x127c0ddc0
ggml_metal_init: loaded kernel_mul_mat_q4_1_f32               0x127c0e020
ggml_metal_init: loaded kernel_mul_mat_q2_K_f32               0x127c0e280
ggml_metal_init: loaded kernel_mul_mat_q3_K_f32               0x127c0e4e0
ggml_metal_init: loaded kernel_mul_mat_q4_K_f32               0x127c0e740
ggml_metal_init: loaded kernel_mul_mat_q5_K_f32               0x127c0e9a0
ggml_metal_init: loaded kernel_mul_mat_q6_K_f32               0x127c0f040
ggml_metal_init: loaded kernel_rope                           0x127c0f560
ggml_metal_init: loaded kernel_alibi_f32                      0x127c0fe20
ggml_metal_init: loaded kernel_cpy_f32_f16                    0x127c106b0
ggml_metal_init: loaded kernel_cpy_f32_f32                    0x127c10f40
ggml_metal_init: loaded kernel_cpy_f16_f16                    0x127c116b0
ggml_metal_init: recommendedMaxWorkingSetSize = 21845.34 MB
ggml_metal_init: hasUnifiedMemory             = true
ggml_metal_init: maxTransferRate              = built-in GPU
llama_new_context_with_model: max tensor size =   102.54 MB
ggml_metal_add_buffer: allocated 'data            ' buffer, size =  3648.31 MB, ( 3648.77 / 21845.34)
ggml_metal_add_buffer: allocated 'eval            ' buffer, size =    10.00 MB, ( 3658.77 / 21845.34)
ggml_metal_add_buffer: allocated 'kv              ' buffer, size =   258.00 MB, ( 3916.77 / 21845.34)
ggml_metal_add_buffer: allocated 'scr0            ' buffer, size =   132.00 MB, ( 4048.77 / 21845.34)
ggml_metal_add_buffer: allocated 'scr1            ' buffer, size =   160.00 MB, ( 4208.77 / 21845.34)
AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | VSX = 0 |

llama_print_timings:        load time =  3358.52 ms
llama_print_timings:      sample time =     0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings: prompt eval time =  3358.23 ms /   137 tokens (   24.51 ms per token,    40.80 tokens per second)
llama_print_timings:        eval time =     0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =  3359.20 ms

llama_print_timings:        load time =  3358.52 ms
llama_print_timings:      sample time =     0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings: prompt eval time =  2740.87 ms /    97 tokens (   28.26 ms per token,    35.39 tokens per second)
llama_print_timings:        eval time =     0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =  2742.21 ms

llama_print_timings:        load time =  3358.52 ms
llama_print_timings:      sample time =     0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings: prompt eval time =  2330.50 ms /    96 tokens (   24.28 ms per token,    41.19 tokens per second)
llama_print_timings:        eval time =     0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =  2331.56 ms

... Cut for brevity

[ggerganov]

Update to latest master - fixed here bf83bff

[MadsRC]

I will try that out immediately!

[MadsRC]

Update to latest master + recompilation (no, I absolutely didn't forget to recompile and was absolutely not wondering why just updating master didn't fix my binary...) worked flawlessly!

Thank you!

[Manouchehri]

@MadsRC Do you still see fairly high CPU load when embedding?

[Goutamxd]

@MadsRC, the issue you noticed—embedding generation relying on the CPU instead of the GPU—was likely due to an older version of llama.cpp. The reason your prompts utilized the GPU while embeddings didn’t is that GPU acceleration for embeddings was not fully implemented in earlier builds. Updating to the latest master branch and recompiling fixed this, as confirmed in your follow-up. If anyone else faces a similar issue, ensure you’re running the latest version and always recompile after updating. 🚀