Skip to content

Commit d77616c

Browse files
hfadzxyhfadzxy
authored
[CI]Add model basic accuracy test(Qwen2.5-0.5B-Instruct) (vllm-project#460)
### What this PR does / why we need it? Add model basic accuracy test(Qwen2.5-0.5B-Instruct) Signed-off-by: hfadzxy <starmoon_zhang@163.com>
1 parent 337edd3 commit d77616c

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+145
-75
lines changed

‎.github/workflows/actionlint.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
#
2-
# Adapted from vllm-project/vllm/blob/main/.github
32
# Copyright 2023 The vLLM team.
43
#
54
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -13,6 +12,7 @@
1312
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1413
# See the License for the specific language governing permissions and
1514
# limitations under the License.
15+
# Adapted from vllm-project/vllm/blob/main/.github
1616
#
1717

1818
name: Lint GitHub Actions workflows

‎.github/workflows/mypy.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
#
2-
# Adapted from vllm-project/vllm/blob/main/.github
32
# Copyright 2023 The vLLM team.
43
#
54
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -13,6 +12,7 @@
1312
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1413
# See the License for the specific language governing permissions and
1514
# limitations under the License.
15+
# Adapted from vllm-project/vllm/blob/main/.github
1616
#
1717

1818
name: mypy

‎.github/workflows/ruff.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
#
2-
# Adapted from vllm-project/vllm/blob/main/.github
32
# Copyright 2023 The vLLM team.
43
#
54
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -13,6 +12,7 @@
1312
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1413
# See the License for the specific language governing permissions and
1514
# limitations under the License.
15+
# Adapted from vllm-project/vllm/blob/main/.github
1616
#
1717

1818
name: ruff

‎.github/workflows/shellcheck.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
#
2-
# Adapted from vllm-project/vllm/blob/main/.github
32
# Copyright 2023 The vLLM team.
43
#
54
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -13,6 +12,7 @@
1312
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1413
# See the License for the specific language governing permissions and
1514
# limitations under the License.
15+
# Adapted from vllm-project/vllm/blob/main/.github
1616
#
1717

1818
name: Lint shell scripts

‎.github/workflows/vllm_ascend_test.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
#
22
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
3-
# This file is a part of the vllm-ascend project.
43
#
54
# Licensed under the Apache License, Version 2.0 (the "License");
65
# you may not use this file except in compliance with the License.
@@ -13,6 +12,7 @@
1312
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1413
# See the License for the specific language governing permissions and
1514
# limitations under the License.
15+
# This file is a part of the vllm-ascend project.
1616
#
1717

1818
name: 'e2e test'

‎.github/workflows/yapf.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
#
2-
# Adapted from vllm-project/vllm/blob/main/.github
32
# Copyright 2023 The vLLM team.
43
#
54
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -13,6 +12,7 @@
1312
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1413
# See the License for the specific language governing permissions and
1514
# limitations under the License.
15+
# Adapted from vllm-project/vllm/blob/main/.github
1616
#
1717

1818
name: yapf

‎Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
#
22
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
3-
# This file is a part of the vllm-ascend project.
43
#
54
# Licensed under the Apache License, Version 2.0 (the "License");
65
# you may not use this file except in compliance with the License.
@@ -13,6 +12,7 @@
1312
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1413
# See the License for the specific language governing permissions and
1514
# limitations under the License.
15+
# This file is a part of the vllm-ascend project.
1616
#
1717

1818
FROM quay.io/ascend/cann:8.0.0-910b-ubuntu22.04-py3.10

‎Dockerfile.openEuler

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
#
22
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
3-
# This file is a part of the vllm-ascend project.
43
#
54
# Licensed under the Apache License, Version 2.0 (the "License");
65
# you may not use this file except in compliance with the License.
@@ -13,6 +12,7 @@
1312
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1413
# See the License for the specific language governing permissions and
1514
# limitations under the License.
15+
# This file is a part of the vllm-ascend project.
1616
#
1717

1818
FROM quay.io/ascend/cann:8.0.0-910b-openeuler22.03-py3.10

‎collect_env.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
#
2-
# Adapted from https://github.com/vllm-project/vllm/blob/main/collect_env.py
32
# Copyright 2023 The vLLM team.
43
#
54
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -13,6 +12,7 @@
1312
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1413
# See the License for the specific language governing permissions and
1514
# limitations under the License.
15+
# Adapted from https://github.com/vllm-project/vllm/blob/main/collect_env.py
1616
#
1717

1818
import datetime

‎docs/source/conf.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
#
22
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
3-
# This file is a part of the vllm-ascend project.
4-
# Adapted from vllm-project/vllm/docs/source/conf.py
53
# Copyright 2023 The vLLM team.
64
#
75
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,6 +13,8 @@
1513
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1614
# See the License for the specific language governing permissions and
1715
# limitations under the License.
16+
# This file is a part of the vllm-ascend project.
17+
# Adapted from vllm-project/vllm/docs/source/conf.py
1818
#
1919

2020
# -- Path setup --------------------------------------------------------------

‎examples/offline_distributed_inference_npu.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
#
22
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
3-
# This file is a part of the vllm-ascend project.
4-
# Adapted from vllm-project/vllm/examples/offline_inference/basic.py
53
# Copyright 2023 The vLLM team.
64
#
75
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,6 +13,8 @@
1513
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1614
# See the License for the specific language governing permissions and
1715
# limitations under the License.
16+
# This file is a part of the vllm-ascend project.
17+
# Adapted from vllm-project/vllm/examples/offline_inference/basic.py
1818
#
1919

2020
from vllm import LLM, SamplingParams

‎examples/offline_inference_audio_language.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
#
22
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
3-
# This file is a part of the vllm-ascend project.
4-
# Adapted from vllm-project/vllm/examples/offline_inference/audio_language.py
53
# Copyright 2023 The vLLM team.
64
#
75
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,6 +13,8 @@
1513
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1614
# See the License for the specific language governing permissions and
1715
# limitations under the License.
16+
# This file is a part of the vllm-ascend project.
17+
# Adapted from vllm-project/vllm/examples/offline_inference/audio_language.py
1818
#
1919
"""
2020
This example shows how to use vLLM for running offline inference

‎examples/offline_inference_npu.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
#
22
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
3-
# This file is a part of the vllm-ascend project.
4-
# Adapted from vllm-project/vllm/examples/offline_inference/basic.py
53
# Copyright 2023 The vLLM team.
64
#
75
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,6 +13,8 @@
1513
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1614
# See the License for the specific language governing permissions and
1715
# limitations under the License.
16+
# This file is a part of the vllm-ascend project.
17+
# Adapted from vllm-project/vllm/examples/offline_inference/basic.py
1818
#
1919

2020
from vllm import LLM, SamplingParams

‎format.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,6 @@
22

33
#
44
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
5-
# This file is a part of the vllm-ascend project.
6-
# Adapted from https://github.com/vllm-project/vllm/tree/main/tools
75
# Copyright 2023 The vLLM team.
86
#
97
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -17,6 +15,8 @@
1715
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1816
# See the License for the specific language governing permissions and
1917
# limitations under the License.
18+
# This file is a part of the vllm-ascend project.
19+
# Adapted from https://github.com/vllm-project/vllm/tree/main/tools
2020
#
2121

2222
# YAPF formatter, adapted from ray and skypilot.

‎mypy.ini

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,5 @@ ignore_missing_imports = True
99
[mypy-transformers.*]
1010
ignore_missing_imports = True
1111

12+
[mypy-lm_eval.*]
13+
ignore_missing_imports = True

‎requirements-dev.txt

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,5 @@
22
modelscope
33
pytest >= 6.0
44
pytest-asyncio
5+
lm-eval
6+
ray

‎setup.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
#
22
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
3-
# This file is a part of the vllm-ascend project.
4-
# Adapted from https://github.com/vllm-project/vllm/blob/main/setup.py
53
# Copyright 2023 The vLLM team.
64
#
75
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,6 +13,8 @@
1513
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1614
# See the License for the specific language governing permissions and
1715
# limitations under the License.
16+
# This file is a part of the vllm-ascend project.
17+
# Adapted from https://github.com/vllm-project/vllm/blob/main/setup.py
1818
#
1919

2020
import importlib.util

‎tests/conftest.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
#
22
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
3-
# This file is a part of the vllm-ascend project.
4-
# Adapted from vllm-project/vllm/blob/main/tests/conftest.py
53
# Copyright 2023 The vLLM team.
64
#
75
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,6 +13,8 @@
1513
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1614
# See the License for the specific language governing permissions and
1715
# limitations under the License.
16+
# This file is a part of the vllm-ascend project.
17+
# Adapted from vllm-project/vllm/blob/main/tests/conftest.py
1818
#
1919

2020
import gc

‎tests/model_utils.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
#
22
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
3-
# This file is a part of the vllm-ascend project.
4-
# Adapted from vllm-project/vllm/blob/main/tests/models/utils.py
53
# Copyright 2023 The vLLM team.
64
#
75
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,6 +13,8 @@
1513
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1614
# See the License for the specific language governing permissions and
1715
# limitations under the License.
16+
# This file is a part of the vllm-ascend project.
17+
# Adapted from vllm-project/vllm/blob/main/tests/models/utils.py
1818
#
1919

2020
import warnings

‎tests/multicard/test_offline_inference_distributed.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
#
22
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
3-
# This file is a part of the vllm-ascend project.
4-
# Adapted from vllm/tests/basic_correctness/test_basic_correctness.py
53
# Copyright 2023 The vLLM team.
64
#
75
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,6 +13,8 @@
1513
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1614
# See the License for the specific language governing permissions and
1715
# limitations under the License.
16+
# This file is a part of the vllm-ascend project.
17+
# Adapted from vllm/tests/basic_correctness/test_basic_correctness.py
1818
#
1919
"""Compare the short outputs of HF and vLLM when using greedy sampling.
2020

‎tests/ops/test_fused_moe.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
11
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
2-
# This file is a part of the vllm-ascend project.
3-
# Adapted from vllm/tests/kernels/test_moe.py
42
# Copyright 2023 The vLLM team.
53
#
64
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,6 +13,8 @@
1513
# See the License for the specific language governing permissions and
1614
# limitations under the License.
1715
# SPDX-License-Identifier: Apache-2.0
16+
# This file is a part of the vllm-ascend project.
17+
# Adapted from vllm/tests/kernels/test_moe.py
1818
"""Tests for the MOE layers.
1919
2020
Run `pytest tests/ops/test_fused_moe.py`.

‎tests/ops/test_rotary_embedding.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
1-
# Adapted from
2-
# https://github.com/vllm-project/vllm/blob/main/vllm/tests/kernels/test_rotary_embedding.py
31
# Copyright 2023 The vLLM team.
42

53
# Copyright (c) Huawei Technologies Co., Ltd. 2024-2025. All rights reserved.
4+
# Adapted from
5+
# https://github.com/vllm-project/vllm/blob/main/vllm/tests/kernels/test_rotary_embedding.py
66

77
from typing import Optional, Tuple, Union
88

‎tests/singlecard/test_accuracy.py

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
#
2+
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
3+
# Copyright 2023 The vLLM team.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
# This file is a part of the vllm-ascend project.
17+
# Adapted from vllm-project/blob/main/tests/entrypoints/llm/test_accuracy.py
18+
#
19+
20+
import gc
21+
import multiprocessing
22+
from multiprocessing import Queue
23+
24+
import lm_eval
25+
import pytest
26+
import torch
27+
28+
# pre-trained model path on Hugging Face.
29+
MODEL_NAME = "Qwen/Qwen2.5-0.5B-Instruct"
30+
# Math reasoning benchmark (Grade School Math 8K).
31+
TASK = "gsm8k"
32+
# Answer validation requiring format consistency.
33+
FILTER = "exact_match,strict-match"
34+
# 3% relative tolerance for numerical accuracy.
35+
RTOL = 0.03
36+
# Baseline accuracy after VLLM optimization.
37+
EXPECTED_VALUE = 0.316
38+
39+
40+
def run_test(queue, more_args=None):
41+
model_args = f"pretrained={MODEL_NAME},max_model_len=4096"
42+
if more_args is not None:
43+
model_args = f"{model_args},{more_args}"
44+
results = lm_eval.simple_evaluate(
45+
model="vllm",
46+
model_args=model_args,
47+
tasks=TASK,
48+
batch_size="auto",
49+
)
50+
result = results["results"][TASK][FILTER]
51+
print("result:", result)
52+
queue.put(result)
53+
del results
54+
torch.npu.empty_cache()
55+
gc.collect()
56+
57+
58+
def test_lm_eval_accuracy(monkeypatch: pytest.MonkeyPatch):
59+
with monkeypatch.context():
60+
result_queue: Queue[float] = multiprocessing.Queue()
61+
p = multiprocessing.Process(target=run_test, args=(result_queue, ))
62+
p.start()
63+
p.join()
64+
result = result_queue.get()
65+
assert (EXPECTED_VALUE - RTOL < result < EXPECTED_VALUE + RTOL), \
66+
f"Expected: {EXPECTED_VALUE}±{RTOL} | Measured: {result}"

0 commit comments

Comments
 (0)