how to run server on GPU?

[moseshu]

my running code is below . but it's not on gpu.?how to run the server on GPU?the output length is short, it seems that n_ctx does't work

python3 -m llama_cpp.server --model ggml-model-f16.bin --port 7777 --host 192.168.0.1 --n_gpu_layers 30 --n_threads 4 --n_ctx 2048

[tomshaffner]

If you check your system resources, do you see it loading on the GPU? If not, you may need to recheck/reinstall the library, as per the readme, to ensure it compiled with GPU support.

[zhangchn]

The output length was limited due to a default parameter max_tokens in each request (which is the json object you send to server from client) being 16.