RuntimeError: linalg.vector_norm: Expected a floating point or complex tensor as input. Got Long

[zaiquanyang]

报错内容如下

[rank0]: Traceback (most recent call last):
[rank0]: File "/mnt/yzq/code/MLLM/RLLM/LLM-Dojo/main_train.py", line 230, in
[rank0]: main()
[rank0]: File "/mnt/yzq/code/MLLM/RLLM/LLM-Dojo/main_train.py", line 217, in main
[rank0]: train_result = trainer.train()
[rank0]: File "/home/yzq/mnt/envs/dojo/lib/python3.10/site-packages/transformers/trainer.py", line 2245, in train
[rank0]: return inner_training_loop(
[rank0]: File "/home/yzq/mnt/envs/dojo/lib/python3.10/site-packages/transformers/trainer.py", line 2560, in _inner_training_loop
[rank0]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank0]: File "/home/yzq/mnt/envs/dojo/lib/python3.10/site-packages/transformers/trainer.py", line 3782, in training_step
[rank0]: self.accelerator.backward(loss, **kwargs)
[rank0]: File "/home/yzq/mnt/envs/dojo/lib/python3.10/site-packages/accelerate/accelerator.py", line 2446, in backward
[rank0]: self.deepspeed_engine_wrapped.backward(loss, **kwargs)
[rank0]: File "/home/yzq/mnt/envs/dojo/lib/python3.10/site-packages/accelerate/utils/deepspeed.py", line 275, in backward
[rank0]: self.engine.step()
[rank0]: File "/home/yzq/mnt/envs/dojo/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2249, in step
[rank0]: self._take_model_step(lr_kwargs)
[rank0]: File "/home/yzq/mnt/envs/dojo/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2152, in _take_model_step
[rank0]: self.optimizer.step()
[rank0]: File "/home/yzq/mnt/envs/dojo/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1852, in step
[rank0]: scaled_global_grad_norm = self.scaled_global_norm()
[rank0]: File "/home/yzq/mnt/envs/dojo/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1799, in scaled_global_norm
[rank0]: return torch.linalg.vector_norm(torch.stack(norm_groups), ord=norm_type)
[rank0]: RuntimeError: linalg.vector_norm: Expected a floating point or complex tensor as input. Got Long
wandb:
wandb: 🚀 View run /home/yzq/mnt/code/MLLM/RLLM/LLM-Dojo/output at: https://wandb.ai/zaiquan/huggingface/runs/auv94sjx
wandb: Find logs at: ../../../../../../../mnt/yzq/code/MLLM/RLLM/LLM-Dojo/wandb/run-20250414_233511-auv94sjx/logs
[rank0]:[W414 23:35:28.191605611 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
[2025-04-14 23:35:28,590] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 777242
[2025-04-14 23:35:28,695] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 777243
[2025-04-14 23:35:28,696] [ERROR] [launch.py:325:sigkill_handler] ['/home/yzq/mnt/envs/dojo/bin/python', '-u', 'main_train.py', '--local_rank=1', '--train_data_path', '/home/yzq/mnt/code/MLLM/RLLM/LLM-Dojo/data/sft_data.jsonl', '--model_name_or_path', 'Qwen/Qwen2-7B-Instruct', '--max_len', '1024', '--num_train_epochs', '1', '--per_device_train_batch_size', '8', '--per_device_eval_batch_size', '1', '--gradient_accumulation_steps', '4', '--task_type', 'sft', '--train_mode', 'qlora', '--output_dir', '/home/yzq/mnt/code/MLLM/RLLM/LLM-Dojo/output', '--save_strategy', 'steps', '--save_steps', '500', '--save_total_limit', '5', '--learning_rate', '2e-4', '--warmup_steps', '10', '--logging_steps', '1', '--lr_scheduler_type', 'cosine_with_min_lr', '--gradient_checkpointing', 'True', '--report_to', 'wandb', '--deepspeed', './train_args/deepspeed_config/ds_config_zero2.json', '--bf16', 'True', '--auto_adapt', 'True'] exits with return code = 1

环境信息如下：

name: dojo
channels:

• defaults
  dependencies:
• _libgcc_mutex=0.1=main
• _openmp_mutex=5.1=1_gnu
• bzip2=1.0.8=h5eee18b_6
• ca-certificates=2025.2.25=h06a4308_0
• ld_impl_linux-64=2.40=h12ee557_0
• libffi=3.4.4=h6a678d5_1
• libgcc-ng=11.2.0=h1234567_1
• libgomp=11.2.0=h1234567_1
• libstdcxx-ng=11.2.0=h1234567_1
• libuuid=1.41.5=h5eee18b_0
• ncurses=6.4=h6a678d5_0
• openssl=3.0.16=h5eee18b_0
• pip=25.0=py310h06a4308_0
• python=3.10.16=he870216_1
• readline=8.2=h5eee18b_0
• setuptools=75.8.0=py310h06a4308_0
• sqlite=3.45.3=h5eee18b_0
• tk=8.6.14=h39e8969_0
• wheel=0.45.1=py310h06a4308_0
• xz=5.6.4=h5eee18b_1
• zlib=1.2.13=h5eee18b_1
• pip:
  • accelerate==1.6.0
  • aiohappyeyeballs==2.6.1
  • aiohttp==3.11.16
  • aiosignal==1.3.2
  • annotated-types==0.7.0
  • async-timeout==5.0.1
  • attrs==25.3.0
  • bitsandbytes==0.45.5
  • certifi==2025.1.31
  • charset-normalizer==3.4.1
  • click==8.1.8
  • datasets==3.5.0
  • deepspeed==0.16.3
  • dill==0.3.8
  • docker-pycreds==0.4.0
  • einops==0.8.1
  • filelock==3.18.0
  • flash-attn==2.7.4.post1
  • frozenlist==1.5.0
  • fsspec==2024.12.0
  • gitdb==4.0.12
  • gitpython==3.1.44
  • hjson==3.1.0
  • huggingface-hub==0.30.2
  • idna==3.10
  • jinja2==3.1.6
  • loguru==0.7.3
  • markdown-it-py==3.0.0
  • markupsafe==3.0.2
  • mdurl==0.1.2
  • mpmath==1.3.0
  • msgpack==1.1.0
  • multidict==6.4.3
  • multiprocess==0.70.16
  • networkx==3.4.2
  • ninja==1.11.1.4
  • numpy==1.26.4
  • nvidia-cublas-cu11==11.11.3.6
  • nvidia-cublas-cu12==12.4.5.8
  • nvidia-cuda-cupti-cu11==11.8.87
  • nvidia-cuda-cupti-cu12==12.4.127
  • nvidia-cuda-nvrtc-cu11==11.8.89
  • nvidia-cuda-nvrtc-cu12==12.4.127
  • nvidia-cuda-runtime-cu11==11.8.89
  • nvidia-cuda-runtime-cu12==12.4.127
  • nvidia-cudnn-cu11==9.1.0.70
  • nvidia-cudnn-cu12==9.1.0.70
  • nvidia-cufft-cu11==10.9.0.58
  • nvidia-cufft-cu12==11.2.1.3
  • nvidia-curand-cu11==10.3.0.86
  • nvidia-curand-cu12==10.3.5.147
  • nvidia-cusolver-cu11==11.4.1.48
  • nvidia-cusolver-cu12==11.6.1.9
  • nvidia-cusparse-cu11==11.7.5.86
  • nvidia-cusparse-cu12==12.3.1.170
  • nvidia-nccl-cu11==2.21.5
  • nvidia-nccl-cu12==2.21.5
  • nvidia-nvjitlink-cu12==12.4.127
  • nvidia-nvtx-cu11==11.8.86
  • nvidia-nvtx-cu12==12.4.127
  • packaging==24.2
  • pandas==2.2.3
  • peft==0.15.1
  • pillow==11.2.1
  • platformdirs==4.3.7
  • propcache==0.3.1
  • protobuf==5.29.4
  • psutil==7.0.0
  • py-cpuinfo==9.0.0
  • pyarrow==19.0.1
  • pydantic==2.11.3
  • pydantic-core==2.33.1
  • pygments==2.19.1
  • python-dateutil==2.9.0.post0
  • pytz==2025.2
  • pyyaml==6.0.2
  • regex==2024.11.6
  • requests==2.32.3
  • rich==14.0.0
  • safetensors==0.5.3
  • sentencepiece==0.2.0
  • sentry-sdk==2.25.1
  • setproctitle==1.3.5
  • six==1.17.0
  • smmap==5.0.2
  • sympy==1.13.1
  • tiktoken==0.9.0
  • tokenizers==0.21.1
  • torch==2.5.0+cu118
  • torchaudio==2.5.0+cu118
  • torchvision==0.20.0+cu118
  • tqdm==4.67.1
  • transformers==4.51.3
  • transformers-stream-generator==0.0.5
  • triton==3.1.0
  • trl==0.16.1
  • typing-extensions==4.13.2
  • typing-inspection==0.4.0
  • tzdata==2025.2
  • urllib3==2.4.0
  • wandb==0.19.9
  • xxhash==3.5.0
  • yarl==1.19.0

这个报错是什么导致的呢，我在Qwen2上尝试做SFT训练

[mst272]

1、ds config里设置overlap_comm=false
2、降低或更新deepspeed=0.15.4 /最新
3、use fp16 than bf16

参考：
1、deepspeedai/DeepSpeed#5241
2、huggingface/transformers#32103

[zaiquanyang]

1、ds config里设置overlap_comm=false 2、降低或更新deepspeed=0.15.4 /最新 3、use fp16 than bf16

参考： 1、deepspeedai/DeepSpeed#5241 2、huggingface/transformers#32103

tks~