Skip to content

Commit 152c36f

Browse files
edlee123edlee123
committed
Reverting back to xeon README.md
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>
1 parent 62f9bf5 commit 152c36f

File tree

1 file changed

+213
-0
lines changed
  • SearchQnA/docker_compose/intel/cpu/xeon

1 file changed

+213
-0
lines changed

‎SearchQnA/docker_compose/intel/cpu/xeon/README.md

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
# Deploying SearchQnA on Intel® Xeon® Processors
2+
3+
This document outlines the single node deployment process for a SearchQnA application utilizing the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservices on Intel Xeon server.
4+
5+
## Table of Contents
6+
7+
1. [SearchQnA Quick Start Deployment](#searchqna-quick-start-deployment)
8+
2. [SearchQnA Docker Compose Files](#searchqna-docker-compose-files)
9+
3. [Validate Microservices](#validate-microservices)
10+
4. [Conclusion](#conclusion)
11+
12+
## SearchQnA Quick Start Deployment
13+
14+
This section describes how to quickly deploy and test the SearchQnA service manually on an Intel® Xeon® processor. The basic steps are:
15+
16+
1. [Access the Code](#access-the-code)
17+
2. [Configure the Deployment Environment](#configure-the-deployment-environment)
18+
3. [Deploy the Services Using Docker Compose](#deploy-the-services-using-docker-compose)
19+
4. [Check the Deployment Status](#check-the-deployment-status)
20+
5. [Validate the Pipeline](#validate-the-pipeline)
21+
6. [Cleanup the Deployment](#cleanup-the-deployment)
22+
23+
### Access the Code
24+
25+
Clone the GenAIExample repository and access the SearchQnA Intel® Xeon® platform Docker Compose files and supporting scripts:
26+
27+
```bash
28+
git clone https://github.com/opea-project/GenAIExamples.git
29+
cd GenAIExamples/SearchQnA
30+
```
31+
32+
Then checkout a released version, such as v1.2:
33+
34+
```bash
35+
git checkout v1.2
36+
```
37+
38+
### Configure the Deployment Environment
39+
40+
To set up environment variables for deploying SearchQnA services, set up some parameters specific to the deployment environment and source the `set_env.sh` script in this directory:
41+
42+
```bash
43+
export host_ip="External_Public_IP" # ip address of the node
44+
export GOOGLE_CSE_ID="your cse id"
45+
export GOOGLE_API_KEY="your google api key"
46+
export HUGGINGFACEHUB_API_TOKEN="Your_HuggingFace_API_Token"
47+
export http_proxy="Your_HTTP_Proxy" # http proxy if any
48+
export https_proxy="Your_HTTPs_Proxy" # https proxy if any
49+
export no_proxy=localhost,127.0.0.1,$host_ip # additional no proxies if needed
50+
export NGINX_PORT=${your_nginx_port} # your usable port for nginx, 80 for example
51+
source ./set_env.sh
52+
```
53+
54+
Consult the section on [SearchQnA Service configuration](#SearchQnA-configuration) for information on how service specific configuration parameters affect deployments.
55+
56+
### Deploy the Services Using Docker Compose
57+
58+
To deploy the SearchQnA services, execute the `docker compose up` command with the appropriate arguments. For a default deployment, execute the command below. It uses the 'compose.yaml' file.
59+
60+
```bash
61+
cd docker_compose/intel/cpu/xeon
62+
docker compose -f compose.yaml up -d
63+
```
64+
65+
> **Note**: developers should build docker image from source when:
66+
>
67+
> - Developing off the git main branch (as the container's ports in the repo may be different > from the published docker image).
68+
> - Unable to download the docker image.
69+
> - Use a specific version of Docker image.
70+
71+
Please refer to the table below to build different microservices from source:
72+
73+
| Microservice | Deployment Guide |
74+
| ------------ | -------------------------------------------------------------------------------------------------- |
75+
| Embedding | [Embedding build guide](https://github.com/opea-project/GenAIComps/tree/main/comps/embeddings/src) |
76+
| Retriever | [Retriever build guide](https://github.com/opea-project/GenAIComps/tree/main/comps/retrievers/src) |
77+
| Reranking | [Reranking build guide](https://github.com/opea-project/GenAIComps/tree/main/comps/rerankings/src) |
78+
| LLM | [LLM build guide](https://github.com/opea-project/GenAIComps/tree/main/comps/llms) |
79+
| MegaService | [MegaService build guide](../../../../README_miscellaneous.md#build-megaservice-docker-image) |
80+
| UI | [Basic UI build guide](../../../../README_miscellaneous.md#build-ui-docker-image) |
81+
82+
### Check the Deployment Status
83+
84+
After running docker compose, check if all the containers launched via docker compose have started:
85+
86+
```bash
87+
docker ps -a
88+
```
89+
90+
For the default deployment, the following containers should have started
91+
92+
If any issues are encountered during deployment, refer to the [Troubleshooting](../../../../README_miscellaneous.md#troubleshooting) section.
93+
94+
### Validate the Pipeline
95+
96+
Once the SearchQnA services are running, test the pipeline using the following command:
97+
98+
```bash
99+
curl http://${host_ip}:3008/v1/searchqna -H "Content-Type: application/json" -d '{
100+
"messages": "What is the latest news? Give me also the source link.",
101+
"stream": "true"
102+
}'
103+
```
104+
105+
**Note** : Access the SearchQnA UI by web browser through this URL: `http://${host_ip}:80`. Please confirm the `80` port is opened in the firewall. To validate each microservice used in the pipeline refer to the [Validate Microservices](#validate-microservices) section.
106+
107+
### Cleanup the Deployment
108+
109+
To stop the containers associated with the deployment, execute the following command:
110+
111+
```bash
112+
docker compose -f compose.yaml down
113+
```
114+
115+
## SearchQnA Docker Compose Files
116+
117+
When deploying a SearchQnA pipeline on an Intel® Xeon® platform, different large language model serving frameworks can be selected. The table below outlines the available configurations included in the application. These configurations can serve as templates and be extended to other components available in [GenAIComps](https://github.com/opea-project/GenAIComps.git).
118+
119+
| File | Description |
120+
| ------------------------------ | --------------------------------------------------------------------------------- |
121+
| [compose.yaml](./compose.yaml) | Default compose file using vllm as serving framework and redis as vector database |
122+
123+
## Validate Microservices
124+
125+
1. Embedding backend Service
126+
127+
```bash
128+
curl http://${host_ip}:3001/embed \
129+
-X POST \
130+
-d '{"inputs":"What is Deep Learning?"}' \
131+
-H 'Content-Type: application/json'
132+
```
133+
134+
2. Embedding Microservice
135+
136+
```bash
137+
curl http://${host_ip}:3002/v1/embeddings\
138+
-X POST \
139+
-d '{"text":"hello"}' \
140+
-H 'Content-Type: application/json'
141+
```
142+
143+
3. Web Retriever Microservice
144+
145+
```bash
146+
export your_embedding=$(python3 -c "import random; embedding = [random.uniform(-1, 1) for _ in range(768)]; print(embedding)")
147+
curl http://${host_ip}:3003/v1/web_retrieval \
148+
-X POST \
149+
-d "{\"text\":\"What is the 2024 holiday schedule?\",\"embedding\":${your_embedding}}" \
150+
-H 'Content-Type: application/json'
151+
```
152+
153+
4. Reranking backend Service
154+
155+
```bash
156+
# TEI Reranking service
157+
curl http://${host_ip}:3004/rerank \
158+
-X POST \
159+
-d '{"query":"What is Deep Learning?", "texts": ["Deep Learning is not...", "Deep learning is..."]}' \
160+
-H 'Content-Type: application/json'
161+
```
162+
163+
5. Reranking Microservice
164+
165+
```bash
166+
curl http://${host_ip}:3005/v1/reranking\
167+
-X POST \
168+
-d '{"initial_query":"What is Deep Learning?", "retrieved_docs": [{"text":"Deep Learning is not..."}, {"text":"Deep learning is..."}]}' \
169+
-H 'Content-Type: application/json'
170+
```
171+
172+
6. LLM backend Service
173+
174+
```bash
175+
# TGI service
176+
curl http://${host_ip}:3006/generate \
177+
-X POST \
178+
-d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":17, "do_sample": true}}' \
179+
-H 'Content-Type: application/json'
180+
```
181+
182+
7. LLM Microservice
183+
184+
```bash
185+
curl http://${host_ip}:3007/v1/chat/completions\
186+
-X POST \
187+
-d '{"query":"What is Deep Learning?","max_tokens":17,"top_k":10,"top_p":0.95,"typical_p":0.95,"temperature":0.01,"repetition_penalty":1.03,"stream":true}' \
188+
-H 'Content-Type: application/json'
189+
```
190+
191+
8. MegaService
192+
193+
```bash
194+
curl http://${host_ip}:3008/v1/searchqna -H "Content-Type: application/json" -d '{
195+
"messages": "What is the latest news? Give me also the source link.",
196+
"stream": "true"
197+
}'
198+
```
199+
200+
9. Nginx Service
201+
202+
```bash
203+
curl http://${host_ip}:${NGINX_PORT}/v1/searchqna \
204+
-H "Content-Type: application/json" \
205+
-d '{
206+
"messages": "What is the latest news? Give me also the source link.",
207+
"stream": "true"
208+
}'
209+
```
210+
211+
## Conclusion
212+
213+
This guide should enable developer to deploy the default configuration or any of the other compose yaml files for different configurations. It also highlights the configurable parameters that can be set before deployment.

0 commit comments

Comments
 (0)