Skip to content

Commit 970b869

Browse files
louie-tsaiwangkl2xiguiw
authored
Add a new section to change LLM model such as deepseek based on validated model table in LLM microservice (#1501)
Signed-off-by: Tsai, Louie <louie.tsai@intel.com> Co-authored-by: Wang, Kai Lawrence <109344418+wangkl2@users.noreply.github.com> Co-authored-by: xiguiw <111278656+xiguiw@users.noreply.github.com>
1 parent 87ff149 commit 970b869

File tree

2 files changed

+31
-0
lines changed

2 files changed

+31
-0
lines changed

ChatQnA/docker_compose/intel/cpu/xeon/README.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,10 +34,22 @@ To set up environment variables for deploying ChatQnA services, follow these ste
3434
```
3535

3636
3. Set up other environment variables:
37+
3738
```bash
3839
source ./set_env.sh
3940
```
4041

42+
4. Change Model for LLM serving
43+
44+
By default, Meta-Llama-3-8B-Instruct is used for LLM serving, the default model can be changed to other validated LLM models.
45+
Please pick a [validated llm models](https://github.com/opea-project/GenAIComps/tree/main/comps/llms/src/text-generation#validated-llm-models) from the table.
46+
To change the default model defined in set_env.sh, overwrite it by exporting LLM_MODEL_ID to the new model or by modifying set_env.sh, and then repeat step 3.
47+
For example, change to Llama-2-7b-chat-hf using the following command.
48+
49+
```bash
50+
export LLM_MODEL_ID="meta-llama/Llama-2-7b-chat-hf"
51+
```
52+
4153
## Quick Start: 2.Run Docker Compose
4254

4355
```bash

ChatQnA/docker_compose/intel/hpu/gaudi/README.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,25 @@ To set up environment variables for deploying ChatQnA services, follow these ste
3939
source ./set_env.sh
4040
```
4141

42+
4. Change Model for LLM serving
43+
44+
By default, Meta-Llama-3-8B-Instruct is used for LLM serving, the default model can be changed to other validated LLM models.
45+
Please pick a [validated llm models](https://github.com/opea-project/GenAIComps/tree/main/comps/llms/src/text-generation#validated-llm-models) from the table.
46+
To change the default model defined in set_env.sh, overwrite it by exporting LLM_MODEL_ID to the new model or by modifying set_env.sh, and then repeat step 3.
47+
For example, change to DeepSeek-R1-Distill-Qwen-32B using the following command.
48+
49+
```bash
50+
export LLM_MODEL_ID="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
51+
```
52+
53+
Please also check [required gaudi cards for different models](https://github.com/opea-project/GenAIComps/tree/main/comps/llms/src/text-generation#system-requirements-for-llm-models) for new models.
54+
It might be necessary to increase the number of Gaudi cards for the model by exporting NUM_CARDS to the new model or by modifying set_env.sh, and then repeating step 3. For example, increase the number of Gaudi cards for DeepSeek-R1-
55+
Distill-Qwen-32B using the following command:
56+
57+
```bash
58+
export NUM_CARDS=4
59+
```
60+
4261
## Quick Start: 2.Run Docker Compose
4362

4463
```bash

0 commit comments

Comments
 (0)