Skip to content

Commit 674c975

Browse files
mhbuehlermhbuehler
authored
Merge branch 'main' into mmqna-audio-query
2 parents f839c43 + 893f324 commit 674c975

File tree

5 files changed

+490
-1
lines changed

5 files changed

+490
-1
lines changed

‎ChatQnA/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -196,7 +196,7 @@ In the below, we provide a table that describes for each microservice component
196196
Gaudi default compose.yaml
197197
| MicroService | Open Source Project | HW | Port | Endpoint |
198198
| ------------ | ------------------- | ----- | ---- | -------------------- |
199-
| Embedding | Langchain | Xeon | 6000 | /v1/embaddings |
199+
| Embedding | Langchain | Xeon | 6000 | /v1/embeddings |
200200
| Retriever | Langchain, Redis | Xeon | 7000 | /v1/retrieval |
201201
| Reranking | Langchain, TEI | Gaudi | 8000 | /v1/reranking |
202202
| LLM | Langchain, TGI | Gaudi | 9000 | /v1/chat/completions |

‎VisualQnA/docker_compose/amd/gpu/rocm/README.md

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# Build Mega Service of VisualQnA on AMD ROCm
2+
3+
This document outlines the deployment process for a VisualQnA application utilizing the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline on Intel Xeon server. The steps include Docker image creation, container deployment via Docker Compose, and service execution to integrate microservices such as `llm`. We will publish the Docker images to Docker Hub soon, it will simplify the deployment process for this service.
4+
5+
## 🚀 Build Docker Images
6+
7+
First of all, you need to build Docker Images locally and install the python package of it.
8+
9+
### 1. Build LVM and NGINX Docker Images
10+
11+
```bash
12+
git clone https://github.com/opea-project/GenAIComps.git
13+
cd GenAIComps
14+
docker build --no-cache -t opea/lvm-tgi:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/lvms/tgi-llava/Dockerfile .
15+
docker build --no-cache -t opea/nginx:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/nginx/Dockerfile .
16+
```
17+
18+
### 2. Build MegaService Docker Image
19+
20+
To construct the Mega Service, we utilize the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline within the `visualqna.py` Python script. Build MegaService Docker image via below command:
21+
22+
```bash
23+
git clone https://github.com/opea-project/GenAIExamples.git
24+
cd GenAIExamples/VisualQnA
25+
docker build --no-cache -t opea/visualqna:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
26+
```
27+
28+
### 3. Build UI Docker Image
29+
30+
Build frontend Docker image via below command:
31+
32+
```bash
33+
cd GenAIExamples/VisualQnA/ui
34+
docker build --no-cache -t opea/visualqna-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f docker/Dockerfile .
35+
```
36+
37+
### 4. Pull TGI AMD ROCm Image
38+
39+
```bash
40+
docker pull ghcr.io/huggingface/text-generation-inference:2.4.1-rocm
41+
```
42+
43+
Then run the command `docker images`, you will have the following 5 Docker Images:
44+
45+
1. `ghcr.io/huggingface/text-generation-inference:2.4.1-rocm`
46+
2. `opea/lvm-tgi:latest`
47+
3. `opea/visualqna:latest`
48+
4. `opea/visualqna-ui:latest`
49+
5. `opea/nginx`
50+
51+
## 🚀 Start Microservices
52+
53+
### Setup Environment Variables
54+
55+
Since the `compose.yaml` will consume some environment variables, you need to setup them in advance as below.
56+
57+
**Export the value of the public IP address of your ROCM server to the `host_ip` environment variable**
58+
59+
> Change the External_Public_IP below with the actual IPV4 value
60+
61+
```
62+
export host_ip="External_Public_IP"
63+
```
64+
65+
**Append the value of the public IP address to the no_proxy list**
66+
67+
```
68+
export your_no_proxy="${your_no_proxy},${host_ip}"
69+
```
70+
71+
```bash
72+
export HOST_IP=${your_host_ip}
73+
export VISUALQNA_TGI_SERVICE_PORT="8399"
74+
export VISUALQNA_HUGGINGFACEHUB_API_TOKEN={your_hugginface_api_token}
75+
export VISUALQNA_CARD_ID="card1"
76+
export VISUALQNA_RENDER_ID="renderD136"
77+
export LVM_MODEL_ID="Xkev/Llama-3.2V-11B-cot"
78+
export MODEL="llava-hf/llava-v1.6-mistral-7b-hf"
79+
export LVM_ENDPOINT="http://${HOST_IP}:8399"
80+
export LVM_SERVICE_PORT=9399
81+
export MEGA_SERVICE_HOST_IP=${HOST_IP}
82+
export LVM_SERVICE_HOST_IP=${HOST_IP}
83+
export BACKEND_SERVICE_ENDPOINT="http://${HOST_IP}:18003/v1/visualqna"
84+
export FRONTEND_SERVICE_IP=${HOST_IP}
85+
export FRONTEND_SERVICE_PORT=18001
86+
export BACKEND_SERVICE_NAME=visualqna
87+
export BACKEND_SERVICE_IP=${HOST_IP}
88+
export BACKEND_SERVICE_PORT=18002
89+
export NGINX_PORT=18003
90+
91+
```
92+
93+
Note: Please replace with `host_ip` with you external IP address, do not use localhost.
94+
95+
Note: You can use set_env.sh file with bash command (. setset_env.sh) to set up needed variables.
96+
97+
### Start all the services Docker Containers
98+
99+
> Before running the docker compose command, you need to be in the folder that has the docker compose yaml file
100+
101+
```bash
102+
cd GenAIExamples/VisualQnA/docker_compose/amd/gpu/rocm
103+
```
104+
105+
```bash
106+
docker compose -f compose.yaml up -d
107+
```
108+
109+
### Validate Microservices
110+
111+
Follow the instructions to validate MicroServices.
112+
113+
> Note: If you see an "Internal Server Error" from the `curl` command, wait a few minutes for the microserver to be ready and then try again.
114+
115+
1. LLM Microservice
116+
117+
```bash
118+
http_proxy="" curl http://${host_ip}:9399/v1/lvm -XPOST -d '{"image": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC", "prompt":"What is this?"}' -H 'Content-Type: application/json'
119+
```
120+
121+
2. MegaService
122+
123+
```bash
124+
curl http://${host_ip}:8888/v1/visualqna -H "Content-Type: application/json" -d '{
125+
"messages": [
126+
{
127+
"role": "user",
128+
"content": [
129+
{
130+
"type": "text",
131+
"text": "What'\''s in this image?"
132+
},
133+
{
134+
"type": "image_url",
135+
"image_url": {
136+
"url": "https://www.ilankelman.org/stopsigns/australia.jpg"
137+
}
138+
}
139+
]
140+
}
141+
],
142+
"max_tokens": 300
143+
}'
144+
```
145+
146+
## 🚀 Launch the UI
147+
148+
To access the frontend, open the following URL in your browser: http://{host_ip}:5173. By default, the UI runs on port 5173 internally. If you prefer to use a different host port to access the frontend, you can modify the port mapping in the `compose.yaml` file as shown below:
149+
150+
```yaml
151+
visualqna-gaudi-ui-server:
152+
image: opea/visualqna-ui:latest
153+
...
154+
ports:
155+
- "80:5173"
156+
```

‎VisualQnA/docker_compose/amd/gpu/rocm/compose.yaml

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# Copyright (C) 2024 Advanced Micro Devices, Inc.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
services:
5+
visualqna-llava-tgi-service:
6+
image: ghcr.io/huggingface/text-generation-inference:2.4.1-rocm
7+
container_name: visualqna-tgi-service
8+
ports:
9+
- "${VISUALQNA_TGI_SERVICE_PORT:-8399}:80"
10+
environment:
11+
no_proxy: ${no_proxy}
12+
http_proxy: ${http_proxy}
13+
https_proxy: ${https_proxy}
14+
TGI_LLM_ENDPOINT: "http://${HOST_IP}:${VISUALQNA_TGI_SERVICE_PORT}"
15+
HUGGINGFACEHUB_API_TOKEN: ${VISUALQNA_HUGGINGFACEHUB_API_TOKEN}
16+
HUGGING_FACE_HUB_TOKEN: ${VISUALQNA_HUGGINGFACEHUB_API_TOKEN}
17+
volumes:
18+
- "/var/opea/visualqna-service/data:/data"
19+
shm_size: 64g
20+
devices:
21+
- /dev/kfd:/dev/kfd
22+
- /dev/dri/:/dev/dri/
23+
cap_add:
24+
- SYS_PTRACE
25+
group_add:
26+
- video
27+
security_opt:
28+
- seccomp:unconfined
29+
ipc: host
30+
command: --model-id ${LVM_MODEL_ID} --max-input-length 4096 --max-total-tokens 8192
31+
lvm-tgi:
32+
image: ${REGISTRY:-opea}/lvm-tgi:${TAG:-latest}
33+
container_name: lvm-tgi-server
34+
depends_on:
35+
- visualqna-llava-tgi-service
36+
ports:
37+
- "9399:9399"
38+
ipc: host
39+
environment:
40+
no_proxy: ${no_proxy}
41+
http_proxy: ${http_proxy}
42+
https_proxy: ${https_proxy}
43+
LVM_ENDPOINT: ${LVM_ENDPOINT}
44+
HF_HUB_DISABLE_PROGRESS_BARS: 1
45+
HF_HUB_ENABLE_HF_TRANSFER: 0
46+
restart: unless-stopped
47+
visualqna-rocm-backend-server:
48+
image: ${REGISTRY:-opea}/visualqna:${TAG:-latest}
49+
container_name: visualqna-rocm-backend-server
50+
depends_on:
51+
- visualqna-llava-tgi-service
52+
- lvm-tgi
53+
ports:
54+
- "${BACKEND_SERVICE_PORT:-8888}:8888"
55+
environment:
56+
- no_proxy=${no_proxy}
57+
- https_proxy=${https_proxy}
58+
- http_proxy=${http_proxy}
59+
- MEGA_SERVICE_HOST_IP=${MEGA_SERVICE_HOST_IP}
60+
- LVM_SERVICE_HOST_IP=${LVM_SERVICE_HOST_IP}
61+
ipc: host
62+
restart: always
63+
visualqna-rocm-ui-server:
64+
image: ${REGISTRY:-opea}/visualqna-ui:${TAG:-latest}
65+
container_name: visualqna-rocm-ui-server
66+
depends_on:
67+
- visualqna-rocm-backend-server
68+
ports:
69+
- "${FRONTEND_SERVICE_PORT:-5173}:5173"
70+
environment:
71+
- no_proxy=${no_proxy}
72+
- https_proxy=${https_proxy}
73+
- http_proxy=${http_proxy}
74+
- BACKEND_BASE_URL=${BACKEND_SERVICE_ENDPOINT}
75+
ipc: host
76+
restart: always
77+
visualqna-nginx-server:
78+
image: ${REGISTRY:-opea}/nginx:${TAG:-latest}
79+
container_name: visualqna-rocm-nginx-server
80+
depends_on:
81+
- visualqna-rocm-backend-server
82+
- visualqna-rocm-ui-server
83+
ports:
84+
- "${NGINX_PORT:-80}:80"
85+
environment:
86+
- no_proxy=${no_proxy}
87+
- https_proxy=${https_proxy}
88+
- http_proxy=${http_proxy}
89+
- FRONTEND_SERVICE_IP=${HOST_IP}
90+
- FRONTEND_SERVICE_PORT=${FRONTEND_SERVICE_PORT}
91+
- BACKEND_SERVICE_NAME=${BACKEND_SERVICE_NAME}
92+
- BACKEND_SERVICE_IP=${HOST_IP}
93+
- BACKEND_SERVICE_PORT=${BACKEND_SERVICE_PORT}
94+
ipc: host
95+
restart: always
96+
97+
networks:
98+
default:
99+
driver: bridge

‎VisualQnA/docker_compose/amd/gpu/rocm/set_env.sh

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/usr/bin/env bash
2+
3+
# Copyright (C) 2024 Advanced Micro Devices, Inc
4+
# SPDX-License-Identifier: Apache-2.0
5+
6+
export HOST_IP=${Your_host_ip_address}
7+
export VISUALQNA_TGI_SERVICE_PORT="8399"
8+
export VISUALQNA_HUGGINGFACEHUB_API_TOKEN=${Your_HUGGINGFACEHUB_API_TOKEN}
9+
export VISUALQNA_CARD_ID="card1"
10+
export VISUALQNA_RENDER_ID="renderD136"
11+
export LVM_MODEL_ID="Xkev/Llama-3.2V-11B-cot"
12+
export LVM_ENDPOINT="http://${HOST_IP}:8399"
13+
export LVM_SERVICE_PORT=9399
14+
export MEGA_SERVICE_HOST_IP=${HOST_IP}
15+
export LVM_SERVICE_HOST_IP=${HOST_IP}
16+
export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:${BACKEND_SERVICE_PORT}/v1/visualqna"
17+
export FRONTEND_SERVICE_IP=${HOST_IP}
18+
export FRONTEND_SERVICE_PORT=18001
19+
export BACKEND_SERVICE_NAME=visualqna
20+
export BACKEND_SERVICE_IP=${HOST_IP}
21+
export BACKEND_SERVICE_PORT=18002
22+
export NGINX_PORT=18003

0 commit comments

Comments
 (0)