Skip to content

Commit ad5523b

Browse files
louie-tsailouie-tsai
authored
Enable OpenTelemtry Tracing for ChatQnA on Xeon and Gaudi by docker compose merge feature (#1488)
Signed-off-by: Louie, Tsai <louie.tsai@intel.com> Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
1 parent 88a8235 commit ad5523b

13 files changed

+181
-30
lines changed

‎ChatQnA/README.md

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,14 @@ cd GenAIExamples/ChatQnA/docker_compose/intel/cpu/xeon/
9191
docker compose up -d
9292
```
9393

94+
To enable Open Telemetry Tracing, compose.telemetry.yaml file need to be merged along with default compose.yaml file.
95+
CPU example with Open Telemetry feature:
96+
97+
```bash
98+
cd GenAIExamples/ChatQnA/docker_compose/intel/cpu/xeon/
99+
docker compose -f compose.yaml -f compose.telemetry.yaml up -d
100+
```
101+
94102
It will automatically download the docker image on `docker hub`:
95103

96104
```bash
@@ -232,6 +240,13 @@ cd GenAIExamples/ChatQnA/docker_compose/intel/hpu/gaudi/
232240
docker compose up -d
233241
```
234242

243+
To enable Open Telemetry Tracing, compose.telemetry.yaml file need to be merged along with default compose.yaml file.
244+
245+
```bash
246+
cd GenAIExamples/ChatQnA/docker_compose/intel/hpu/gaudi/
247+
docker compose -f compose.yaml -f compose.telemetry.yaml up -d
248+
```
249+
235250
Refer to the [Gaudi Guide](./docker_compose/intel/hpu/gaudi/README.md) to build docker images from source.
236251

237252
### Deploy ChatQnA on Xeon
@@ -243,6 +258,13 @@ cd GenAIExamples/ChatQnA/docker_compose/intel/cpu/xeon/
243258
docker compose up -d
244259
```
245260

261+
To enable Open Telemetry Tracing, compose.telemetry.yaml file need to be merged along with default compose.yaml file.
262+
263+
```bash
264+
cd GenAIExamples/ChatQnA/docker_compose/intel/cpu/xeon/
265+
docker compose -f compose.yaml -f compose.telemetry.yaml up -d
266+
```
267+
246268
Refer to the [Xeon Guide](./docker_compose/intel/cpu/xeon/README.md) for more instructions on building docker images from source.
247269

248270
### Deploy ChatQnA on NVIDIA GPU
@@ -346,7 +368,7 @@ OPEA microservice deployment can easily be monitored through Grafana dashboards
346368

347369
## Tracing Services with OpenTelemetry Tracing and Jaeger
348370

349-
> NOTE: limited support. Only LLM inference serving with TGI on Gaudi is enabled for this feature.
371+
> NOTE: This feature is disabled by default. Please check the Deploy ChatQnA sessions for how to enable this feature with compose_telemetry.yaml file.
350372
351373
OPEA microservice and TGI/TEI serving can easily be traced through Jaeger dashboards in conjunction with OpenTelemetry Tracing feature. Follow the [README](https://github.com/opea-project/GenAIComps/tree/main/comps/cores/telemetry#tracing) to trace additional functions if needed.
352374

‎ChatQnA/docker_compose/intel/cpu/xeon/README.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,14 @@ To set up environment variables for deploying ChatQnA services, follow these ste
4444
docker compose up -d
4545
```
4646

47+
To enable Open Telemetry Tracing, compose.telemetry.yaml file need to be merged along with default compose.yaml file.
48+
CPU example with Open Telemetry feature:
49+
50+
```bash
51+
cd GenAIExamples/ChatQnA/docker_compose/intel/cpu/xeon/
52+
docker compose -f compose.yaml -f compose.telemetry.yaml up -d
53+
```
54+
4755
It will automatically download the docker image on `docker hub`:
4856

4957
```bash
@@ -263,12 +271,16 @@ If use vLLM as the LLM serving backend.
263271
docker compose -f compose.yaml up -d
264272
# Start ChatQnA without Rerank Pipeline
265273
docker compose -f compose_without_rerank.yaml up -d
274+
# Start ChatQnA with Rerank Pipeline and Open Telemetry Tracing
275+
docker compose -f compose.yaml -f compose.telemetry.yaml up -d
266276
```
267277

268278
If use TGI as the LLM serving backend.
269279

270280
```bash
271281
docker compose -f compose_tgi.yaml up -d
282+
# Start ChatQnA with Open Telemetry Tracing
283+
docker compose -f compose_tgi.yaml -f compose_tgi.telemetry.yaml up -d
272284
```
273285

274286
### Validate Microservices

‎ChatQnA/docker_compose/intel/cpu/xeon/compose.telemetry.yaml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
services:
5+
tei-embedding-service:
6+
command: --model-id ${EMBEDDING_MODEL_ID} --auto-truncate --otlp-endpoint $OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
7+
tei-reranking-service:
8+
command: --model-id ${RERANK_MODEL_ID} --auto-truncate --otlp-endpoint $OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
9+
jaeger:
10+
image: jaegertracing/all-in-one:latest
11+
container_name: jaeger
12+
ports:
13+
- "16686:16686"
14+
- "4317:4317"
15+
- "4318:4318"
16+
- "9411:9411"
17+
ipc: host
18+
environment:
19+
no_proxy: ${no_proxy}
20+
http_proxy: ${http_proxy}
21+
https_proxy: ${https_proxy}
22+
COLLECTOR_ZIPKIN_HOST_PORT: 9411
23+
restart: unless-stopped
24+
chatqna-xeon-backend-server:
25+
environment:
26+
- ENABLE_OPEA_TELEMETRY=true
27+
- TELEMETRY_ENDPOINT=${TELEMETRY_ENDPOINT}

‎ChatQnA/docker_compose/intel/cpu/xeon/compose_tgi.telemetry.yaml

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
services:
5+
tei-embedding-service:
6+
command: --model-id ${EMBEDDING_MODEL_ID} --auto-truncate --otlp-endpoint $OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
7+
tei-reranking-service:
8+
command: --model-id ${RERANK_MODEL_ID} --auto-truncate --otlp-endpoint $OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
9+
tgi-service:
10+
command: --model-id ${LLM_MODEL_ID} --cuda-graphs 0 --otlp-endpoint $OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
11+
jaeger:
12+
image: jaegertracing/all-in-one:latest
13+
container_name: jaeger
14+
ports:
15+
- "16686:16686"
16+
- "4317:4317"
17+
- "4318:4318"
18+
- "9411:9411"
19+
ipc: host
20+
environment:
21+
no_proxy: ${no_proxy}
22+
http_proxy: ${http_proxy}
23+
https_proxy: ${https_proxy}
24+
COLLECTOR_ZIPKIN_HOST_PORT: 9411
25+
restart: unless-stopped
26+
chatqna-xeon-backend-server:
27+
environment:
28+
- ENABLE_OPEA_TELEMETRY=true
29+
- TELEMETRY_ENDPOINT=${TELEMETRY_ENDPOINT}

‎ChatQnA/docker_compose/intel/cpu/xeon/set_env.sh

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,3 +19,7 @@ export INDEX_NAME="rag-redis"
1919
# Set it as a non-null string, such as true, if you want to enable logging facility,
2020
# otherwise, keep it as "" to disable it.
2121
export LOGFLAG=""
22+
# Set OpenTelemetry Tracing Endpoint
23+
export JAEGER_IP=$(ip route get 8.8.8.8 | grep -oP 'src \K[^ ]+')
24+
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://$JAEGER_IP:4317
25+
export TELEMETRY_ENDPOINT=http://$JAEGER_IP:4318/v1/traces

‎ChatQnA/docker_compose/intel/hpu/gaudi/README.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,12 @@ To set up environment variables for deploying ChatQnA services, follow these ste
4545
docker compose up -d
4646
```
4747

48+
To enable Open Telemetry Tracing, compose.telemetry.yaml file need to be merged along with default compose.yaml file.
49+
50+
```bash
51+
docker compose -f compose.yaml -f compose.telemetry.yaml up -d
52+
```
53+
4854
It will automatically download the docker image on `docker hub`:
4955

5056
```bash
@@ -259,12 +265,16 @@ If use vLLM as the LLM serving backend.
259265
docker compose -f compose.yaml up -d
260266
# Start ChatQnA without Rerank Pipeline
261267
docker compose -f compose_without_rerank.yaml up -d
268+
# Start ChatQnA with Rerank Pipeline and Open Telemetry Tracing
269+
docker compose -f compose.yaml -f compose.telemetry.yaml up -d
262270
```
263271

264272
If use TGI as the LLM serving backend.
265273

266274
```bash
267275
docker compose -f compose_tgi.yaml up -d
276+
# Start ChatQnA with Open Telemetry Tracing
277+
docker compose -f compose_tgi.yaml -f compose_tgi.telemetry.yaml up -d
268278
```
269279

270280
If you want to enable guardrails microservice in the pipeline, please follow the below command instead:

‎ChatQnA/docker_compose/intel/hpu/gaudi/compose.telemetry.yaml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
services:
5+
tei-embedding-service:
6+
command: --model-id ${EMBEDDING_MODEL_ID} --auto-truncate --otlp-endpoint $OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
7+
tei-reranking-service:
8+
command: --model-id ${RERANK_MODEL_ID} --auto-truncate --otlp-endpoint $OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
9+
jaeger:
10+
image: jaegertracing/all-in-one:latest
11+
container_name: jaeger
12+
ports:
13+
- "16686:16686"
14+
- "4317:4317"
15+
- "4318:4318"
16+
- "9411:9411"
17+
ipc: host
18+
environment:
19+
no_proxy: ${no_proxy}
20+
http_proxy: ${http_proxy}
21+
https_proxy: ${https_proxy}
22+
COLLECTOR_ZIPKIN_HOST_PORT: 9411
23+
restart: unless-stopped
24+
chatqna-gaudi-backend-server:
25+
environment:
26+
- ENABLE_OPEA_TELEMETRY=true
27+
- TELEMETRY_ENDPOINT=${TELEMETRY_ENDPOINT}

‎ChatQnA/docker_compose/intel/hpu/gaudi/compose_tgi.telemetry.yaml

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
services:
5+
tei-embedding-service:
6+
command: --model-id ${EMBEDDING_MODEL_ID} --auto-truncate --otlp-endpoint $OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
7+
tei-reranking-service:
8+
command: --model-id ${RERANK_MODEL_ID} --auto-truncate --otlp-endpoint $OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
9+
tgi-service:
10+
command: --model-id ${LLM_MODEL_ID} --max-input-length 2048 --max-total-tokens 4096 --otlp-endpoint $OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
11+
jaeger:
12+
image: jaegertracing/all-in-one:latest
13+
container_name: jaeger
14+
ports:
15+
- "16686:16686"
16+
- "4317:4317"
17+
- "4318:4318"
18+
- "9411:9411"
19+
ipc: host
20+
environment:
21+
no_proxy: ${no_proxy}
22+
http_proxy: ${http_proxy}
23+
https_proxy: ${https_proxy}
24+
COLLECTOR_ZIPKIN_HOST_PORT: 9411
25+
restart: unless-stopped
26+
chatqna-gaudi-backend-server:
27+
environment:
28+
- ENABLE_OPEA_TELEMETRY=true
29+
- TELEMETRY_ENDPOINT=${TELEMETRY_ENDPOINT}

‎ChatQnA/docker_compose/intel/hpu/gaudi/compose_tgi.yaml

Lines changed: 3 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,6 @@ services:
2525
INDEX_NAME: ${INDEX_NAME}
2626
TEI_ENDPOINT: http://tei-embedding-service:80
2727
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
28-
TELEMETRY_ENDPOINT: ${TELEMETRY_ENDPOINT}
2928
tei-embedding-service:
3029
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5
3130
container_name: tei-embedding-gaudi-server
@@ -38,7 +37,7 @@ services:
3837
no_proxy: ${no_proxy}
3938
http_proxy: ${http_proxy}
4039
https_proxy: ${https_proxy}
41-
command: --model-id ${EMBEDDING_MODEL_ID} --auto-truncate --otlp-endpoint $OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
40+
command: --model-id ${EMBEDDING_MODEL_ID} --auto-truncate
4241
retriever:
4342
image: ${REGISTRY:-opea}/retriever:${TAG:-latest}
4443
container_name: retriever-redis-server
@@ -56,7 +55,6 @@ services:
5655
INDEX_NAME: ${INDEX_NAME}
5756
TEI_EMBEDDING_ENDPOINT: http://tei-embedding-service:80
5857
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
59-
TELEMETRY_ENDPOINT: ${TELEMETRY_ENDPOINT}
6058
LOGFLAG: ${LOGFLAG}
6159
RETRIEVER_COMPONENT_NAME: "OPEA_RETRIEVER_REDIS"
6260
restart: unless-stopped
@@ -80,7 +78,7 @@ services:
8078
HABANA_VISIBLE_DEVICES: all
8179
OMPI_MCA_btl_vader_single_copy_mechanism: none
8280
MAX_WARMUP_SEQUENCE_LENGTH: 512
83-
command: --model-id ${RERANK_MODEL_ID} --auto-truncate --otlp-endpoint $OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
81+
command: --model-id ${RERANK_MODEL_ID} --auto-truncate
8482
tgi-service:
8583
image: ghcr.io/huggingface/tgi-gaudi:2.0.6
8684
container_name: tgi-gaudi-server
@@ -106,22 +104,7 @@ services:
106104
cap_add:
107105
- SYS_NICE
108106
ipc: host
109-
command: --model-id ${LLM_MODEL_ID} --num-shard ${NUM_CARDS} --max-input-length 2048 --max-total-tokens 4096 --otlp-endpoint $OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
110-
jaeger:
111-
image: jaegertracing/all-in-one:latest
112-
container_name: jaeger
113-
ports:
114-
- "16686:16686"
115-
- "4317:4317"
116-
- "4318:4318"
117-
- "9411:9411"
118-
ipc: host
119-
environment:
120-
no_proxy: ${no_proxy}
121-
http_proxy: ${http_proxy}
122-
https_proxy: ${https_proxy}
123-
COLLECTOR_ZIPKIN_HOST_PORT: 9411
124-
restart: unless-stopped
107+
command: --model-id ${LLM_MODEL_ID} --num-shard ${NUM_CARDS} --max-input-length 2048 --max-total-tokens 4096
125108
chatqna-gaudi-backend-server:
126109
image: ${REGISTRY:-opea}/chatqna:${TAG:-latest}
127110
container_name: chatqna-gaudi-backend-server
@@ -147,7 +130,6 @@ services:
147130
- LLM_SERVER_PORT=${LLM_SERVER_PORT:-80}
148131
- LLM_MODEL=${LLM_MODEL_ID}
149132
- LOGFLAG=${LOGFLAG}
150-
- TELEMETRY_ENDPOINT=${TELEMETRY_ENDPOINT}
151133
ipc: host
152134
restart: always
153135
chatqna-gaudi-ui-server:

‎ChatQnA/tests/test_compose_on_gaudi.sh

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,9 +49,12 @@ function start_services() {
4949
export INDEX_NAME="rag-redis"
5050
export HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN}
5151
export host_ip=${ip_address}
52+
export JAEGER_IP=$(ip route get 8.8.8.8 | grep -oP 'src \K[^ ]+')
53+
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://$JAEGER_IP:4317
54+
export TELEMETRY_ENDPOINT=http://$JAEGER_IP:4318/v1/traces
5255

5356
# Start Docker Containers
54-
docker compose -f compose.yaml up -d > ${LOG_PATH}/start_services_with_compose.log
57+
docker compose -f compose.yaml -f compose_telemetry.yaml up -d > ${LOG_PATH}/start_services_with_compose.log
5558
n=0
5659
until [[ "$n" -ge 160 ]]; do
5760
echo "n=$n"
@@ -172,7 +175,7 @@ function validate_frontend() {
172175

173176
function stop_docker() {
174177
cd $WORKPATH/docker_compose/intel/hpu/gaudi
175-
docker compose -f compose.yaml down
178+
docker compose -f compose.yaml -f compose_telemetry.yaml down
176179
}
177180

178181
function main() {

‎ChatQnA/tests/test_compose_on_xeon.sh

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,9 +49,12 @@ function start_services() {
4949
export INDEX_NAME="rag-redis"
5050
export HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN}
5151
export host_ip=${ip_address}
52+
export JAEGER_IP=$(ip route get 8.8.8.8 | grep -oP 'src \K[^ ]+')
53+
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://$JAEGER_IP:4317
54+
export TELEMETRY_ENDPOINT=http://$JAEGER_IP:4318/v1/traces
5255

5356
# Start Docker Containers
54-
docker compose -f compose.yaml up -d > ${LOG_PATH}/start_services_with_compose.log
57+
docker compose -f compose.yaml -f compose_telemetry.yaml up -d > ${LOG_PATH}/start_services_with_compose.log
5558
n=0
5659
until [[ "$n" -ge 100 ]]; do
5760
docker logs vllm-service > ${LOG_PATH}/vllm_service_start.log 2>&1
@@ -172,7 +175,7 @@ function validate_frontend() {
172175

173176
function stop_docker() {
174177
cd $WORKPATH/docker_compose/intel/cpu/xeon
175-
docker compose -f compose.yaml down
178+
docker compose -f compose.yaml -f compose_telemetry.yaml down
176179
}
177180

178181
function main() {

‎ChatQnA/tests/test_compose_tgi_on_gaudi.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ function start_services() {
5454
export TELEMETRY_ENDPOINT=http://$JAEGER_IP:4318/v1/traces
5555

5656
# Start Docker Containers
57-
docker compose -f compose_tgi.yaml up -d > ${LOG_PATH}/start_services_with_compose.log
57+
docker compose -f compose_tgi.yaml -f compose_tgi_telemetry.yaml up -d > ${LOG_PATH}/start_services_with_compose.log
5858

5959
n=0
6060
until [[ "$n" -ge 500 ]]; do
@@ -218,7 +218,7 @@ function validate_frontend() {
218218

219219
function stop_docker() {
220220
cd $WORKPATH/docker_compose/intel/hpu/gaudi
221-
docker compose -f compose_tgi.yaml down
221+
docker compose -f compose_tgi.yaml -f compose_tgi_telemetry.yaml down
222222
}
223223

224224
function main() {

‎ChatQnA/tests/test_compose_tgi_on_xeon.sh

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,9 +48,12 @@ function start_services() {
4848
export LLM_MODEL_ID="meta-llama/Meta-Llama-3-8B-Instruct"
4949
export INDEX_NAME="rag-redis"
5050
export HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN}
51+
export JAEGER_IP=$(ip route get 8.8.8.8 | grep -oP 'src \K[^ ]+')
52+
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://$JAEGER_IP:4317
53+
export TELEMETRY_ENDPOINT=http://$JAEGER_IP:4318/v1/traces
5154

5255
# Start Docker Containers
53-
docker compose -f compose_tgi.yaml up -d > ${LOG_PATH}/start_services_with_compose.log
56+
docker compose -f compose_tgi.yaml -f compose_tgi_telemetry.yaml up -d > ${LOG_PATH}/start_services_with_compose.log
5457

5558
n=0
5659
until [[ "$n" -ge 100 ]]; do
@@ -216,7 +219,7 @@ function validate_frontend() {
216219

217220
function stop_docker() {
218221
cd $WORKPATH/docker_compose/intel/cpu/xeon
219-
docker compose -f compose_tgi.yaml down
222+
docker compose -f compose_tgi.yaml -f compose_tgi_telemetry.yaml down
220223
}
221224

222225
function main() {

0 commit comments

Comments
 (0)