[CI] Refactor CI #952

wangxiyuan · 2025-05-26T02:51:04Z

remove some useless test func and file
fix format.sh problem
enable full test for singlecard and multicard
move long term test to long_term folder. For this kind of test, it only runs by labeled and daily test. Include: spec decode、accuracy test

After refactor:

There are 4 test modules

singlecard: contains the test running on one NPU. It'll be run for each PR and daily test.
multicard: contains the test running on multi NPUs. It'll be run for each PR and daily test.
long_term: contains the test that cost much time(Now include spec decode and accuracy test). It'll be run for the PR with long-term-test labeled and daily test.
e2e: contains the test for doc and pd feature. It'll be run for the PR with pd-test labeled and daily test.

Todo:

some test are skipped, they should be fixed and reenabled in the future.
pyhccl test for multicard doesn't work at all. It should be enabled as well.
ensure long-term-test pass by daily test.

Know issue

Now, ready-for-test labels is required to start pd test or long term test. And when long-term-test or pd-test is labeled after another one, the old labeled test will be re-run again. So the labeled test should be ran in the following step:

decide which test need run, then label it. long-term-test or pd-test or both.
add ready-for-test label, then the test will be ran.

wangxiyuan · 2025-05-27T03:05:13Z

.github/workflows/vllm_ascend_test.yaml

-            pytest -sv tests/singlecard/test_ilama_lora.py
-            pytest -sv tests/ops
-            pytest -sv tests/compile
+            # AscendScheduler doesn't work, fix it later


this should be fixed by @zzzzwwjj in #939

wangxiyuan · 2025-05-27T03:05:27Z

.github/workflows/vllm_ascend_test.yaml

-            pytest -sv tests/compile
+            # AscendScheduler doesn't work, fix it later
+            # pytest -sv tests/singlecard/tets_schedule.py
+            # guided decoding doesn't work, fix it later


this should be fixed by @shen-shanshan in #969

tests/singlecard/test_camem.py

wangxiyuan · 2025-05-27T03:07:28Z

tests/singlecard/sample/test_rejection_sampler.py

@@ -322,6 +322,7 @@ def test_deterministic_when_seeded(
                assert torch.equal(results[j][i], results[0][i])


+@pytest.mark.skipif(True, reason="Test failed, need fix")


this should be fixed by @ponix-j

wangxiyuan · 2025-05-27T06:37:46Z

tests/singlecard/test_camem.py

@@ -59,9 +57,9 @@ def test_basic_camem():
    assert torch.allclose(output, torch.ones_like(output) * 3)


+@pytest.mark.skipif(True, reason="test failed, should be fixed later")


This should be fixed by @Potabk

wangxiyuan · 2025-05-27T06:39:46Z

.github/workflows/vllm_ascend_test.yaml

            pytest -sv tests/singlecard/test_offline_inference.py
-            pytest -sv tests/ops
+            # AscendScheduler doesn't work, fix it later


ditto @zzzzwwjj

wangxiyuan · 2025-05-27T06:40:02Z

.github/workflows/vllm_ascend_test.yaml

-            pytest -sv tests/ops
+            # AscendScheduler doesn't work, fix it later
+            # pytest -sv tests/singlecard/tets_schedule.py
+            # guided decoding doesn't work, fix it later


ditto @shen-shanshan

wangxiyuan · 2025-05-27T07:47:46Z

tests/multicard/test_offline_inference_distributed.py

-if __name__ == "__main__":
-    import pytest
-    pytest.main([__file__])
+@pytest.mark.skipif(os.getenv("VLLM_USE_V1") == "1",


This should be fixed by @MengqingCao when v1 is ready

Yikun

LGTM if CI passed

.github/workflows/vllm_ascend_test_long_term.yaml

Yikun · 2025-05-27T09:53:08Z

.github/workflows/vllm_ascend_test_long_term.yaml

+          # spec decode test
+          VLLM_USE_MODELSCOPE=true pytest -sv tests/long_term/spec_decode/e2e/test_v1_spec_decode.py
+          VLLM_USE_MODELSCOPE=True pytest -sv tests/long_term/spec_decode/e2e/test_mtp_correctness.py  # it needs a clean process
+          pytest -sv tests/long_term/spec_decode --ignore=tests/long_term/spec_decode/e2e/test_mtp_correctness.py --ignore=tests/long_term/spec_decode/e2e/test_v1_spec_decode.py


also cc @mengwei805 the spec decode will be triggered by manaully after this PR.

See the commit message:

1. decide which test need run, then label it. It can be `long-term-test` or `pd-test` or both. 2. add `ready-for-test` label, then the test will be ran.

i consider it is a wonderful refactor

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

1. remove some useless test func and file 2. fix format.sh problem 3. enable full test for singlecard and multicard 4. move long term test to long_term folder. For this kind of test, it only runs by labeled and daily test. Include: spec decode、accuracy test There are 4 test modules - `singlecard`: contains the test running on one NPU. It'll be run for each PR and daily test. - `multicard`: contains the test running on multi NPUs. It'll be run for each PR and daily test. - `long_term`: contains the test that cost much time(Now include `spec decode` and `accuracy` test). It'll be run for the PR with `long-term-test` labeled and daily test. - `e2e`: contains the test for doc and pd feature. It'll be run for the PR with `pd-test` labeled and daily test. 1. some test are skipped, they should be fixed and reenabled in the future. 2. pyhccl test for multicard doesn't work at all. It should be enabled as well. 3. ensure long-term-test pass by daily test. Now, `ready` labels is required to start pd test or long term test. And when `long-term-test` or `pd-test` is labeled after another one, the old labeled test will be re-run again. So the labeled test should be ran in the following step: 1. decide which test need run, then label it. `long-term-test` or `pd-test` or both. 2. add `ready-for-test` label, then the test will be ran. Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com> Signed-off-by: hfadzxy <starmoon_zhang@163.com>

github-actions bot added the module:tests label May 26, 2025

wangxiyuan force-pushed the refactor_ci branch 2 times, most recently from 475f9bb to 74bcd9c Compare May 26, 2025 08:20

github-actions bot added the module:core label May 26, 2025

wangxiyuan force-pushed the refactor_ci branch 3 times, most recently from f183ad6 to 1d51c1e Compare May 27, 2025 02:36

github-actions bot removed the module:core label May 27, 2025

wangxiyuan force-pushed the refactor_ci branch from 1d51c1e to d4538c6 Compare May 27, 2025 03:03

wangxiyuan commented May 27, 2025

View reviewed changes

Potabk mentioned this pull request May 27, 2025

[CI] Add nightly CI #926

Closed

wangxiyuan force-pushed the refactor_ci branch 4 times, most recently from 8bad69b to 13c3d01 Compare May 27, 2025 06:31

wangxiyuan commented May 27, 2025

View reviewed changes

wangxiyuan force-pushed the refactor_ci branch from 13c3d01 to afdd79a Compare May 27, 2025 06:40

wangxiyuan mentioned this pull request May 27, 2025

[aclgraph] implentment NPUPiecewiseBackend to enable aclgraph #836

Merged

wangxiyuan force-pushed the refactor_ci branch from afdd79a to d956799 Compare May 27, 2025 07:45

wangxiyuan commented May 27, 2025

View reviewed changes

wangxiyuan added long-term-test enable long term test for PR pd-test enable pd test for PR and removed long-term-test enable long term test for PR labels May 27, 2025

wangxiyuan force-pushed the refactor_ci branch from d956799 to 039523a Compare May 27, 2025 08:05

wangxiyuan added pd-test enable pd test for PR long-term-test enable long term test for PR and removed long-term-test enable long term test for PR pd-test enable pd test for PR labels May 27, 2025

wangxiyuan added ready read for review and removed pd-test enable pd test for PR labels May 27, 2025

wangxiyuan force-pushed the refactor_ci branch from 039523a to 347262c Compare May 27, 2025 09:10

wangxiyuan added pd-test enable pd test for PR ready read for review and removed ready read for review long-term-test enable long term test for PR pd-test enable pd test for PR labels May 27, 2025

wangxiyuan force-pushed the refactor_ci branch from 347262c to a59242c Compare May 27, 2025 09:38

Yikun approved these changes May 27, 2025

View reviewed changes

wangxiyuan force-pushed the refactor_ci branch from a59242c to 487c8e4 Compare May 27, 2025 12:14

[CI] Refactor CI

2b8f812

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

wangxiyuan force-pushed the refactor_ci branch from 487c8e4 to 2b8f812 Compare May 27, 2025 14:33

wangxiyuan merged commit e2a0c19 into vllm-project:main May 27, 2025
15 checks passed

shen-shanshan mentioned this pull request May 28, 2025

[Bugfix][CI] Update guided decoding backend list #969

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[CI] Refactor CI #952

[CI] Refactor CI #952

Uh oh!

wangxiyuan commented May 26, 2025 •

edited

Loading

Uh oh!

wangxiyuan May 27, 2025 •

edited

Loading

Uh oh!

wangxiyuan May 27, 2025 •

edited

Loading

Uh oh!

Uh oh!

wangxiyuan May 27, 2025

Uh oh!

wangxiyuan May 27, 2025

Uh oh!

wangxiyuan May 27, 2025

Uh oh!

wangxiyuan May 27, 2025

Uh oh!

wangxiyuan May 27, 2025

Uh oh!

Yikun left a comment

Uh oh!

Uh oh!

Yikun May 27, 2025

Uh oh!

wangxiyuan May 27, 2025

Uh oh!

mengwei805 May 28, 2025

Uh oh!

Uh oh!

Uh oh!

		@@ -322,6 +322,7 @@ def test_deterministic_when_seeded(
		assert torch.equal(results[j][i], results[0][i])


		@pytest.mark.skipif(True, reason="Test failed, need fix")

		@@ -59,9 +57,9 @@ def test_basic_camem():
		assert torch.allclose(output, torch.ones_like(output) * 3)


		@pytest.mark.skipif(True, reason="test failed, should be fixed later")

[CI] Refactor CI #952

[CI] Refactor CI #952

Uh oh!

Conversation

wangxiyuan commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

After refactor:

Todo:

Know issue

Uh oh!

wangxiyuan May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wangxiyuan May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yikun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wangxiyuan commented May 26, 2025 •

edited

Loading

wangxiyuan May 27, 2025 •

edited

Loading

wangxiyuan May 27, 2025 •

edited

Loading