Skip to content

[Bug]: CUDA error: unknown error when running vllm serve on WSL2 Ubuntu22.04 #19077

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
1 task done
ezioasche opened this issue Jun 3, 2025 · 0 comments
Open
1 task done

[Bug]: CUDA error: unknown error when running vllm serve on WSL2 Ubuntu22.04 #19077

ezioasche opened this issue Jun 3, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@ezioasche
Copy link

Your current environment

The output of python collect_env.py 
🐧 root 19:12:36 06/03/25 ξ‚° ❲c❳ conda311-clean ξ‚° 🏠 ξ‚° βœ… ξ‚° python -u /mnt/d/collect_env.py
INFO 06-03 19:16:45 __init__.py:183] Automatically detected platform cuda.
Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : version 3.22.1
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.5.1+cu124
Is debug build               : False
CUDA used to build PyTorch   : 12.4
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.11.11 (main, Dec 11 2024, 16:28:39) [GCC 11.2.0] (64-bit runtime)
Python platform              : Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.4.131
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration :
GPU 0: NVIDIA GeForce RTX 4080 SUPER
GPU 1: NVIDIA GeForce RTX 4080 SUPER

Nvidia driver version        : 576.52
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   46 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          16
On-line CPU(s) list:             0-15
Vendor ID:                       GenuineIntel
Model name:                      Intel(R) Core(TM) i7-7820X CPU @ 3.60GHz
CPU family:                      6
Model:                           85
Thread(s) per core:              2
Core(s) per socket:              8
Socket(s):                       1
Stepping:                        4
BogoMIPS:                        7199.99
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves flush_l1d arch_capabilities
Hypervisor vendor:               Microsoft
Virtualization type:             full
L1d cache:                       256 KiB (8 instances)
L1i cache:                       256 KiB (8 instances)
L2 cache:                        8 MiB (8 instances)
L3 cache:                        11 MiB (1 instance)
Vulnerability Itlb multihit:     KVM: Mitigation: VMX unsupported
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown

==============================
Versions of relevant libraries
==============================
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.575.51
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.4.0
[pip3] torch==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.52.4
[pip3] triton==3.1.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.5.147               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi
[conda] nvidia-ml-py              12.575.51                pypi_0    pypi
[conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi
[conda] pyzmq                     26.4.0                   pypi_0    pypi
[conda] torch                     2.5.1                    pypi_0    pypi
[conda] torchvision               0.20.1                   pypi_0    pypi
[conda] transformers              4.52.4                   pypi_0    pypi
[conda] triton                    3.1.0                    pypi_0    pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
Neuron SDK Version           : N/A
vLLM Version                 : 0.7.0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
        GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS                             N/A
GPU1    SYS      X                              N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
LD_LIBRARY_PATH=/root/conda311-clean/lib/python3.11/site-packages/cv2/../../lib64::/usr/local/cuda-12.4/lib64
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐧 root 19:16:53 06/03/25 ξ‚° ❲c❳ conda311-clean ξ‚° 🏠 ξ‚° βœ… ξ‚°

πŸ› Describe the bug

I installed Ubuntu22.04 for WSL2 on Windows10 21H2(19044.3086) and followed the steps in the NVIDA documentation to install the driver.

Then I executed 'vllm serve' (Refer to the output information below for parameters) in python==3.11 version of the conda environment after pip install vllm==0.7.0 (Tsinghua source) .

Then, "CUDA error: unknown error" is thrown before loading module.

The output of run vllm serve /root/modules/QwQ-32B/ --trust-remote-code --served-model-name qwq --tensor-parallel-size 2 --pipeline-parallel-size 1 --max-model-len 1024 --gpu-memory-utilization 0.95 --max-num-seqs 256 --disable-custom-all-reduce 
🐧 root 19:07:11 06/03/25 ξ‚° ❲c❳ conda311-clean ξ‚° 🏠 ξ‚° βœ… ξ‚° vllm serve /root/modules/QwQ-32B/ --trust-remote-code --served-model-name qwq --tensor-parallel-size 2 --pipeline-parallel-size 1 --max-model-len 1024 --gpu-memory-utilization 0.95 --max-num-seqs 256 --disable-custom-all-reduce
INFO 06-03 19:07:23 __init__.py:183] Automatically detected platform cuda.
INFO 06-03 19:07:24 api_server.py:835] vLLM API server version 0.7.0
INFO 06-03 19:07:24 api_server.py:836] args: Namespace(subparser='serve', model_tag='/root/modules/QwQ-32B/', config='', host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/root/modules/QwQ-32B/', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=1024, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=True, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['qwq'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function serve at 0x7fd924fe2660>)
WARNING 06-03 19:07:24 utils.py:2069] Found ulimit of 1024 and failed to automatically increasewith error current limit exceeds maximum limit. This can cause fd limit errors like`OSError: [Errno 24] Too many open files`. Consider increasing with ulimit -n
INFO 06-03 19:07:24 api_server.py:203] Started engine process with PID 9596
INFO 06-03 19:07:27 __init__.py:183] Automatically detected platform cuda.
INFO 06-03 19:07:29 config.py:520] This model supports multiple tasks: {'reward', 'classify', 'score', 'embed', 'generate'}. Defaulting to 'generate'.
INFO 06-03 19:07:29 config.py:1328] Defaulting to use mp for distributed inference
INFO 06-03 19:07:33 config.py:520] This model supports multiple tasks: {'generate', 'embed', 'classify', 'reward', 'score'}. Defaulting to 'generate'.
INFO 06-03 19:07:33 config.py:1328] Defaulting to use mp for distributed inference
INFO 06-03 19:07:33 llm_engine.py:232] Initializing an LLM engine (v0.7.0) with config: model='/root/modules/QwQ-32B/', speculative_config=None, tokenizer='/root/modules/QwQ-32B/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=1024, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=qwq, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
WARNING 06-03 19:07:33 multiproc_worker_utils.py:298] Reducing Torch parallelism from 8 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 06-03 19:07:33 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
WARNING 06-03 19:07:34 interface.py:275] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
INFO 06-03 19:07:34 cuda.py:225] Using Flash Attention backend.
INFO 06-03 19:07:36 __init__.py:183] Automatically detected platform cuda.
(VllmWorkerProcess pid=9680) INFO 06-03 19:07:37 multiproc_worker_utils.py:227] Worker ready; awaiting tasks
(VllmWorkerProcess pid=9680) WARNING 06-03 19:07:38 interface.py:275] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
(VllmWorkerProcess pid=9680) INFO 06-03 19:07:38 cuda.py:225] Using Flash Attention backend.
INFO 06-03 19:07:41 utils.py:938] Found nccl from library libnccl.so.2
INFO 06-03 19:07:41 pynccl.py:67] vLLM is using nccl==2.21.5
(VllmWorkerProcess pid=9680) INFO 06-03 19:07:41 utils.py:938] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=9680) INFO 06-03 19:07:41 pynccl.py:67] vLLM is using nccl==2.21.5
INFO 06-03 19:07:41 shm_broadcast.py:256] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_63b33934'), local_subscribe_port=54095, remote_subscribe_port=None)
INFO 06-03 19:07:42 model_runner.py:1110] Starting to load model /root/modules/QwQ-32B/...
(VllmWorkerProcess pid=9680) INFO 06-03 19:07:42 model_runner.py:1110] Starting to load model /root/modules/QwQ-32B/...
ERROR 06-03 19:07:43 engine.py:387] CUDA error: unknown error
ERROR 06-03 19:07:43 engine.py:387] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
ERROR 06-03 19:07:43 engine.py:387] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
ERROR 06-03 19:07:43 engine.py:387] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
ERROR 06-03 19:07:43 engine.py:387] Traceback (most recent call last):
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 378, in run_mp_engine
ERROR 06-03 19:07:43 engine.py:387]     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 06-03 19:07:43 engine.py:387]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 121, in from_engine_args
ERROR 06-03 19:07:43 engine.py:387]     return cls(ipc_path=ipc_path,
ERROR 06-03 19:07:43 engine.py:387]            ^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 73, in __init__
ERROR 06-03 19:07:43 engine.py:387]     self.engine = LLMEngine(*args, **kwargs)
ERROR 06-03 19:07:43 engine.py:387]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 271, in __init__
ERROR 06-03 19:07:43 engine.py:387]     self.model_executor = executor_class(vllm_config=vllm_config, )
ERROR 06-03 19:07:43 engine.py:387]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 260, in __init__
ERROR 06-03 19:07:43 engine.py:387]     super().__init__(*args, **kwargs)
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 49, in __init__
ERROR 06-03 19:07:43 engine.py:387]     self._init_executor()
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/executor/mp_distributed_executor.py", line 123, in _init_executor
ERROR 06-03 19:07:43 engine.py:387]     self._run_workers("load_model",
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/executor/mp_distributed_executor.py", line 183, in _run_workers
ERROR 06-03 19:07:43 engine.py:387]     driver_worker_output = run_method(self.driver_worker, sent_method,
ERROR 06-03 19:07:43 engine.py:387]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/utils.py", line 2208, in run_method
ERROR 06-03 19:07:43 engine.py:387]     return func(*args, **kwargs)
ERROR 06-03 19:07:43 engine.py:387]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/worker/worker.py", line 182, in load_model
ERROR 06-03 19:07:43 engine.py:387]     self.model_runner.load_model()
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1112, in load_model
ERROR 06-03 19:07:43 engine.py:387]     self.model = get_model(vllm_config=self.vllm_config)
ERROR 06-03 19:07:43 engine.py:387]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
ERROR 06-03 19:07:43 engine.py:387]     return loader.load_model(vllm_config=vllm_config)
ERROR 06-03 19:07:43 engine.py:387]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/model_loader/loader.py", line 376, in load_model
ERROR 06-03 19:07:43 engine.py:387]     model = _initialize_model(vllm_config=vllm_config)
ERROR 06-03 19:07:43 engine.py:387]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/model_loader/loader.py", line 118, in _initialize_model
ERROR 06-03 19:07:43 engine.py:387]     return model_class(vllm_config=vllm_config, prefix=prefix)
ERROR 06-03 19:07:43 engine.py:387]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 451, in __init__
ERROR 06-03 19:07:43 engine.py:387]     self.model = Qwen2Model(vllm_config=vllm_config,
ERROR 06-03 19:07:43 engine.py:387]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/compilation/decorators.py", line 149, in __init__
ERROR 06-03 19:07:43 engine.py:387]     old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 305, in __init__
ERROR 06-03 19:07:43 engine.py:387]     self.start_layer, self.end_layer, self.layers = make_layers(
ERROR 06-03 19:07:43 engine.py:387]                                                     ^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 555, in make_layers
ERROR 06-03 19:07:43 engine.py:387]     [PPMissingLayer() for _ in range(start_layer)] + [
ERROR 06-03 19:07:43 engine.py:387]                                                      ^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 556, in <listcomp>
ERROR 06-03 19:07:43 engine.py:387]     maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
ERROR 06-03 19:07:43 engine.py:387]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 307, in <lambda>
ERROR 06-03 19:07:43 engine.py:387]     lambda prefix: Qwen2DecoderLayer(config=config,
ERROR 06-03 19:07:43 engine.py:387]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 218, in __init__
ERROR 06-03 19:07:43 engine.py:387]     self.mlp = Qwen2MLP(
ERROR 06-03 19:07:43 engine.py:387]                ^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 73, in __init__
ERROR 06-03 19:07:43 engine.py:387]     self.gate_up_proj = MergedColumnParallelLinear(
ERROR 06-03 19:07:43 engine.py:387]                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/layers/linear.py", line 433, in __init__
ERROR 06-03 19:07:43 engine.py:387]     super().__init__(input_size=input_size,
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/layers/linear.py", line 311, in __init__
ERROR 06-03 19:07:43 engine.py:387]     self.quant_method.create_weights(
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/layers/linear.py", line 127, in create_weights
ERROR 06-03 19:07:43 engine.py:387]     weight = Parameter(torch.empty(sum(output_partition_sizes),
ERROR 06-03 19:07:43 engine.py:387]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387]   File "/root/conda311-clean/lib/python3.11/site-packages/torch/utils/_device.py", line 106, in __torch_function__
ERROR 06-03 19:07:43 engine.py:387]     return func(*args, **kwargs)
ERROR 06-03 19:07:43 engine.py:387]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 06-03 19:07:43 engine.py:387] RuntimeError: CUDA error: unknown error
ERROR 06-03 19:07:43 engine.py:387] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
ERROR 06-03 19:07:43 engine.py:387] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
ERROR 06-03 19:07:43 engine.py:387] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
ERROR 06-03 19:07:43 engine.py:387]
Process SpawnProcess-1:
ERROR 06-03 19:07:43 multiproc_worker_utils.py:122] Worker VllmWorkerProcess pid 9680 died, exit code: -15
INFO 06-03 19:07:43 multiproc_worker_utils.py:126] Killing local vLLM worker processes
Traceback (most recent call last):
  File "/root/conda311-clean/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/conda311-clean/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 389, in run_mp_engine
    raise e
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 378, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 121, in from_engine_args
    return cls(ipc_path=ipc_path,
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 73, in __init__
    self.engine = LLMEngine(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 271, in __init__
    self.model_executor = executor_class(vllm_config=vllm_config, )
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 260, in __init__
    super().__init__(*args, **kwargs)
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 49, in __init__
    self._init_executor()
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/executor/mp_distributed_executor.py", line 123, in _init_executor
    self._run_workers("load_model",
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/executor/mp_distributed_executor.py", line 183, in _run_workers
    driver_worker_output = run_method(self.driver_worker, sent_method,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/utils.py", line 2208, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/worker/worker.py", line 182, in load_model
    self.model_runner.load_model()
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1112, in load_model
    self.model = get_model(vllm_config=self.vllm_config)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
    return loader.load_model(vllm_config=vllm_config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/model_loader/loader.py", line 376, in load_model
    model = _initialize_model(vllm_config=vllm_config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/model_loader/loader.py", line 118, in _initialize_model
    return model_class(vllm_config=vllm_config, prefix=prefix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 451, in __init__
    self.model = Qwen2Model(vllm_config=vllm_config,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/compilation/decorators.py", line 149, in __init__
    old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 305, in __init__
    self.start_layer, self.end_layer, self.layers = make_layers(
                                                    ^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 555, in make_layers
    [PPMissingLayer() for _ in range(start_layer)] + [
                                                     ^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 556, in <listcomp>
    maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 307, in <lambda>
    lambda prefix: Qwen2DecoderLayer(config=config,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 218, in __init__
    self.mlp = Qwen2MLP(
               ^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 73, in __init__
    self.gate_up_proj = MergedColumnParallelLinear(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/layers/linear.py", line 433, in __init__
    super().__init__(input_size=input_size,
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/layers/linear.py", line 311, in __init__
    self.quant_method.create_weights(
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/model_executor/layers/linear.py", line 127, in create_weights
    weight = Parameter(torch.empty(sum(output_partition_sizes),
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/torch/utils/_device.py", line 106, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: unknown error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[rank0]:[W603 19:07:44.976751991 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
Traceback (most recent call last):
  File "/root/conda311-clean/bin/vllm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/scripts.py", line 201, in main
    args.dispatch_function(args)
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/scripts.py", line 42, in serve
    uvloop.run(run_server(args))
  File "/root/conda311-clean/lib/python3.11/site-packages/uvloop/__init__.py", line 105, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/root/conda311-clean/lib/python3.11/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 863, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/root/conda311-clean/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 133, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/root/conda311-clean/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda311-clean/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 227, in build_async_engine_client_from_engine_args
    raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.
/root/conda311-clean/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
 🐧 root 19:07:51 06/03/25 ξ‚° ❲c❳ conda311-clean ξ‚° 🏠 ξ‚° ❌1 ξ‚°

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
@ezioasche ezioasche added the bug Something isn't working label Jun 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ezioasche