From df7c34a0c27fe89e707f34241db76822a85cbcb6 Mon Sep 17 00:00:00 2001 From: Jiaxin Shan Date: Mon, 8 Jul 2024 16:27:32 -0700 Subject: [PATCH 1/2] Fix the lora adapter path in server startup script --- docs/source/models/lora.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/models/lora.rst b/docs/source/models/lora.rst index 934887a607a..47b2fb5a13c 100644 --- a/docs/source/models/lora.rst +++ b/docs/source/models/lora.rst @@ -64,7 +64,7 @@ LoRA adapted models can also be served with the Open-AI compatible vLLM server. python -m vllm.entrypoints.openai.api_server \ --model meta-llama/Llama-2-7b-hf \ --enable-lora \ - --lora-modules sql-lora=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/ + --lora-modules sql-lora=$HOME/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/snapshots/0dfa347e8877a4d4ed19ee56c140fa518470028c/ The server entrypoint accepts all other LoRA configuration parameters (``max_loras``, ``max_lora_rank``, ``max_cpu_loras``, etc.), which will apply to all forthcoming requests. Upon querying the ``/models`` endpoint, we should see our LoRA along From 776ee2f4373f7046f8fe39e0e56da1cebb960da8 Mon Sep 17 00:00:00 2001 From: Jiaxin Shan Date: Tue, 9 Jul 2024 16:41:34 -0700 Subject: [PATCH 2/2] Add a comment on the potential change about model commit id --- docs/source/models/lora.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/source/models/lora.rst b/docs/source/models/lora.rst index 47b2fb5a13c..5cc3076073f 100644 --- a/docs/source/models/lora.rst +++ b/docs/source/models/lora.rst @@ -66,6 +66,9 @@ LoRA adapted models can also be served with the Open-AI compatible vLLM server. --enable-lora \ --lora-modules sql-lora=$HOME/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/snapshots/0dfa347e8877a4d4ed19ee56c140fa518470028c/ +.. note:: + The commit ID `0dfa347e8877a4d4ed19ee56c140fa518470028c` may change over time. Please check the latest commit ID in your environment to ensure you are using the correct one. + The server entrypoint accepts all other LoRA configuration parameters (``max_loras``, ``max_lora_rank``, ``max_cpu_loras``, etc.), which will apply to all forthcoming requests. Upon querying the ``/models`` endpoint, we should see our LoRA along with its base model: