[Bug]: Seed issue with Pipeline Parallel

[andoorve]

Your current environment

v0.5.1

🐛 Describe the bug

OpenAI API specifies that you can provide a seed: https://platform.openai.com/docs/api-reference/chat/create#chat-create-seed

This allows reproducibility for example with non-zero temperature parameter.

Currently, any state information is stored/advanced on the driver process only. We need to extend this to the worker actually doing the sampling.

[binxuan]

• 1, got openai.InternalServerError: Internal Server Error when I specified seed in the completion request

[andoorve]

@binxuan do you have any other debug information? That shouldn't be expected

[binxuan]

Below is my error trace, not sure if this is the root cause.

Traceback (most recent call last):
File "python/ray/_raylet.pyx", line 919, in ray._raylet.prepare_args_internal
File "/home/binxuan/.conda/envs/vllm_dist/lib/python3.10/site-packages/ray/_private/serialization.py", line 519, in serialize
return self._serialize_to_msgpack(value)
File "/home/binxuan/.conda/envs/vllm_dist/lib/python3.10/site-packages/ray/_private/serialization.py", line 497, in _serialize_to_msgpack
pickle5_serialized_object = self._serialize_to_pickle5(
File "/home/binxuan/.conda/envs/vllm_dist/lib/python3.10/site-packages/ray/_private/serialization.py", line 444, in _serialize_to_pickle5
raise e
File "/home/binxuan/.conda/envs/vllm_dist/lib/python3.10/site-packages/ray/_private/serialization.py", line 439, in _serialize_to_pickle5
inband = pickle.dumps(
File "/home/binxuan/.conda/envs/vllm_dist/lib/python3.10/site-packages/ray/cloudpickle/cloudpickle.py", line 1479, in dumps
cp.dump(obj)
File "/home/binxuan/.conda/envs/vllm_dist/lib/python3.10/site-packages/ray/cloudpickle/cloudpickle.py", line 1245, in dump
return super().dump(obj)
TypeError: cannot pickle 'torch._C.Generator' object

[andoorve]

@njhill any comments? Did you run into the above?

[njhill]

@andoorve yes this is the known issue, I should hopefully have time to fix it tomorrow.

[sekh77]

Hi @njhill - Hope you have had some chance to fix this issue. Is it available now in the latest version (0.5.2)?

[njhill]

@sekh77 it's not in 0.5.2. I am working on it right now and it should be ready today.

[sekh77]

@njhill - Ok, thanks. Will this also be available for people who are on 0.5.1?

[njhill]

@sekh77 yes, it will be available to everyone! You'll just have to upgrade to the latest version (hopefully will be in a new release in the next day or two).