GPU support

[draco1544]

It’s possible to make the webui and the container work with a version of llama that use gpu ?

[voarsh2]

Here's hoping. Consumers would only be able to use 8/12 GB's of VRAM though.

[gaby]

This would be something to ask the llama.cpp team, until they don't add support for GPU's we can't add it to Serge.

[edricus]

This would be something to ask the llama.cpp team, until they don't add support for GPU's we can't add it to Serge.

It seems it's now officially supported ggml-org/llama.cpp#1827

[gaby]

This would be something to ask the llama.cpp team, until they don't add support for GPU's we can't add it to Serge.

It seems it's now officially supported ggerganov/llama.cpp#1827

That's a draft, it's not merged/released yet

[roperscrossroads]

@gaby

I'm not familiar with this project but I saw this issue mentioned in the llama.cpp pull request. llama.cpp already has cuBLAS and CLBlast support in the master branch. The PR mentioned above improves performance for GPUs with CUDA.

Check out the docs where they mention building with LLAMA_OPENBLAS and LLAMA_CUBLAS enabled for more information.

[rakuri255]

Seems this issue should be open

[gaby]

Duplicated of #43