Skip to content

Commit c65d7d4

Browse files
authored
fix vllm output in chatqna (#1038)
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com>
1 parent 9d12416 commit c65d7d4

File tree

3 files changed

+4
-8
lines changed

3 files changed

+4
-8
lines changed

ChatQnA/chatqna.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -166,7 +166,10 @@ def align_generator(self, gen, **kwargs):
166166
try:
167167
# sometimes yield empty chunk, do a fallback here
168168
json_data = json.loads(json_str)
169-
if json_data["choices"][0]["finish_reason"] != "eos_token":
169+
if (
170+
json_data["choices"][0]["finish_reason"] != "eos_token"
171+
and "content" in json_data["choices"][0]["delta"]
172+
):
170173
yield f"data: {repr(json_data['choices'][0]['delta']['content'].encode('utf-8'))}\n\n"
171174
except Exception as e:
172175
yield f"data: {repr(json_str.encode('utf-8'))}\n\n"

ChatQnA/docker_image_build/build.yaml

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -83,12 +83,6 @@ services:
8383
dockerfile: comps/llms/text-generation/vllm/langchain/dependency/Dockerfile.intel_hpu
8484
extends: chatqna
8585
image: ${REGISTRY:-opea}/llm-vllm-hpu:${TAG:-latest}
86-
llm-vllm-ray:
87-
build:
88-
context: GenAIComps
89-
dockerfile: comps/llms/text-generation/vllm/ray/Dockerfile
90-
extends: chatqna
91-
image: ${REGISTRY:-opea}/llm-vllm-ray:${TAG:-latest}
9286
llm-vllm-ray-hpu:
9387
build:
9488
context: GenAIComps

docker_images_list.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,6 @@ Take ChatQnA for example. ChatQnA is a chatbot application service based on the
8080
| [opea/llm-vllm-hpu](https://hub.docker.com/r/opea/llm-vllm-hpu) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/llms/text-generation/vllm/langchain/dependency/Dockerfile.intel_hpu) | The docker image exposed the OPEA LLM microservice upon vLLM docker image for use by GenAI apps on the Gaudi |
8181
| [opea/llm-vllm-llamaindex](https://hub.docker.com/r/opea/llm-vllm-llamaindex) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/llms/text-generation/vllm/llama_index/Dockerfile) | This docker image exposes OPEA LLM microservices to the llamaindex framework's vLLM Docker image for use by GenAI applications |
8282
| [opea/llm-vllm-llamaindex-hpu](https://hub.docker.com/r/opea/llm-vllm-llamaindex-hpu) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/llms/text-generation/vllm/llama_index/dependency/Dockerfile.intel_hpu) | This docker image exposes OPEA LLM microservices to the llamaindex framework's vLLM Docker image for use by GenAI applications on the gaudi |
83-
| [opea/llm-vllm-ray](https://hub.docker.com/r/opea/llm-vllm-ray) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/llms/text-generation/vllm/ray/Dockerfile) | The docker image exposes the OPEA LLM microservices Ray-based upon the vLLM Docker image for GenAI application use |
8483
| [opea/llm-vllm-ray-hpu](https://hub.docker.com/r/opea/llm-vllm-ray-hpu) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/llms/text-generation/vllm/ray/dependency/Dockerfile) | The docker image exposes Ray-based OPEA LLM microservices upon the vLLM Docker image for use by GenAI applications on the Gaudi |
8584
| [opea/llava-hpu](https://hub.docker.com/r/opea/llava-hpu) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/lvms/llava/dependency/Dockerfile.intel_hpu) | The docker image exposed the OPEA microservice running LLaVA as a large visual model (LVM) service for GenAI application use on the Gaudi |
8685
| [opea/lvm-tgi](https://hub.docker.com/r/opea/lvm-tgi) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/lvms/tgi-llava/Dockerfile) | This docker image is designed to build a large visual model (LVM) microservice using the HuggingFace Text Generation Inference(TGI) framework. The microservice accepts document input and generates a answer to question. |

0 commit comments

Comments
 (0)