GPU Layers for No Model mode.

[razvanab]

Describe the Issue
Gamma 2 with Ollama run slowly with the default GPU Layers in No Model mode.
Please add a setting to change the GPU Layers when running in No Model mode

Additional Information:

koboldcpp-1.86

Windows 11
CPU: Ryzen 2700x
RAM: 32GB
GPU: Nvidia RTX 3050
VRAM: 8GB

[LostRuins]

Does it not predict the layers on your device? What does the estimate show? Do you have a screenshot of your GUI launcher

[razvanab]

I use it with the nomodel switch, which doesn't load the koboldcpp GUI.

koboldcpp_cu12.exe --nomodel

[razvanab]

I found out that you can launch the GUI with --showgui even with --nomodel. I guess changing GPU layers on the fly is not yet supported.

[LostRuins]

Oh well that makes a lot of sense. How can it estimate gpu layers without a model lol. It has no idea what you want to use, so how will it predict the layers to use.

[henk717]

Just don't use ollama, our own backend is much more suitable for our UI.
I made a script here that can liberate ollama models and provide you with the GGUF : https://gist.github.com/henk717/4ffd6b7486d2cb1e74fb8251ec46d56e

[razvanab]

Oh, nice, thanks.