You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -21,3 +22,89 @@ Below series are NOT supported yet:
21
22
- Ascend 910, Ascend 910 Pro B (Ascend-cann-kernels-910) unplanned yet
22
23
23
24
From a technical view, vllm-ascend support would be possible if the torch-npu is supported. Otherwise, we have to implement it by using custom ops. We are also welcome to join us to improve together.
25
+
26
+
### 2. How to get our docker containers?
27
+
28
+
You can get our containers at `Quay.io`, e.g., [<u>vllm-ascend</u>](https://quay.io/repository/ascend/vllm-ascend?tab=tags) and [<u>cann</u>](https://quay.io/repository/ascend/cann?tab=tags).
29
+
30
+
If you are in China, you can use `daocloud` to accelerate your downloading:
Find more details [<u>here</u>](https://vllm-ascend.readthedocs.io/en/latest/user_guide/supported_models.html).
41
+
42
+
### 4. How to get in touch with our community?
43
+
44
+
There are many channels that you can communicate with our community developers / users:
45
+
46
+
- Submit a GitHub [<u>issue</u>](https://github.com/vllm-project/vllm-ascend/issues?page=1).
47
+
- Join our [<u>weekly meeting</u>](https://docs.google.com/document/d/1hCSzRTMZhIB8vRq1_qOOjx4c9uYUxvdQvDsMV2JcSrw/edit?tab=t.0#heading=h.911qu8j8h35z) and share your ideas.
48
+
- Join our [<u>WeChat</u>](https://github.com/vllm-project/vllm-ascend/issues/227) group and ask your quenstions.
49
+
- Join our ascend channel in [<u>vLLM forums</u>](https://discuss.vllm.ai/c/hardware-support/vllm-ascend-support/6) and publish your topics.
50
+
51
+
### 5. What features does vllm-ascend V1 supports?
52
+
53
+
Find more details [<u>here</u>](https://github.com/vllm-project/vllm-ascend/issues/414).
54
+
55
+
### 6. How to solve the problem of "Failed to infer device type" or "libatb.so: cannot open shared object file"?
56
+
57
+
Basically, the reason is that the NPU environment is not configured correctly. You can:
58
+
1. try `source /usr/local/Ascend/nnal/atb/set_env.sh` to enable NNAL package.
59
+
2. try `source /usr/local/Ascend/ascend-toolkit/set_env.sh` to enable CANN package.
60
+
3. try `npu-smi info` to check whether the NPU is working.
61
+
62
+
If all above steps are not working, you can try the following code with python to check whether there is any error:
63
+
64
+
```
65
+
import torch
66
+
import torch_npu
67
+
import vllm
68
+
```
69
+
70
+
If all above steps are not working, feel free to submit a GitHub issue.
71
+
72
+
### 7. How does vllm-ascend perform?
73
+
74
+
Currently, only some models are improved. Such as `Qwen2 VL`, `Deepseek V3`. Others are not good enough. In the future, we will support graph mode and custom ops to improve the performance of vllm-ascend. And when the official release of vllm-ascend is released, you can install `mindie-turbo` with `vllm-ascend` to speed up the inference as well.
75
+
76
+
### 8. How vllm-ascend work with vllm?
77
+
vllm-ascend is a plugin for vllm. Basically, the version of vllm-ascend is the same as the version of vllm. For example, if you use vllm 0.7.3, you should use vllm-ascend 0.7.3 as well. For main branch, we will make sure `vllm-ascend` and `vllm` are compatible by each commit.
78
+
79
+
### 9. Does vllm-ascend support Prefill Disaggregation feature?
80
+
81
+
Currently, only 1P1D is supported by vllm. For vllm-ascend, it'll be done by [this PR](https://github.com/vllm-project/vllm-ascend/pull/432). For NPND, vllm is not stable and fully supported yet. We will make it stable and supported by vllm-ascend in the future.
82
+
83
+
### 10. Does vllm-ascend support quantization method?
84
+
85
+
Currently, w8a8 quantization is already supported by vllm-ascend originally on v0.8.4rc2 or heigher, If you're using vllm 0.7.3 version, w8a8 quantization is supporeted with the integration of vllm-ascend and mindie-turbo, please use `pip install vllm-ascend[mindie-turbo]`.
86
+
87
+
### 11. How to run w8a8 DeepSeek model?
88
+
89
+
Currently, w8a8 DeepSeek is working in process: [support AscendW8A8 quantization](https://github.com/vllm-project/vllm-ascend/pull/511)
90
+
91
+
Please run DeepSeek with BF16 now, follwing the [Multi-Node DeepSeek inferencing tutorail](https://vllm-ascend.readthedocs.io/en/main/tutorials/multi_node.html)
92
+
93
+
### 12. There is not output in log when loading models using vllm-ascend, How to solve it?
94
+
95
+
If you're using vllm 0.7.3 version, this is a known progress bar display issue in VLLM, which has been resolved in [this PR](https://github.com/vllm-project/vllm/pull/12428), please cherry-pick it locally by yourself. Otherwise, please fill up an issue.
96
+
97
+
### 13. How vllm-ascend is tested
98
+
99
+
vllm-ascend is tested by functional test, performance test and accuracy test.
100
+
101
+
-**Functional test**: we added CI, includes portion of vllm's native unit tests and vllm-ascend's own unit tests，on vllm-ascend's test, we test basic functionality、popular models availability and [supported features](https://vllm-ascend.readthedocs.io/en/latest/user_guide/suppoted_features.html) via e2e test
102
+
103
+
-**Performance test**: we provide [benchmark](https://github.com/vllm-project/vllm-ascend/tree/main/benchmarks) tools for end-to-end performance benchmark which can easily to re-route locally, we'll publish a perf website like [vllm](https://simon-mo-workspace.observablehq.cloud/vllm-dashboard-v0/perf) does to show the performance test results for each pull request
104
+
105
+
-**Accuracy test**: we're working on adding accuracy test to CI as well.
106
+
107
+
Finnall, for each release, we'll publish the performance test and accuracy test report in the future.
108
+
109
+
### 14. How to fix the error "InvalidVersion" when using vllm-ascend?
110
+
It's usually because you have installed an dev/editable version of vLLM package. In this case, we provide the env variable `VLLM_VERSION` to let users specify the version of vLLM package to use. Please set the env variable `VLLM_VERSION` to the version of vLLM package you have installed. The format of `VLLM_VERSION` should be `X.Y.Z`.
0 commit comments