Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

以ray模式启动，当内存不足的时候,会溢写到磁盘吗？ #574

Open
3 tasks done
javapythonphp opened this issue Feb 11, 2025 · 2 comments
Open
3 tasks done

以ray模式启动，当内存不足的时候,会溢写到磁盘吗？ #574

javapythonphp opened this issue Feb 11, 2025 · 2 comments
Labels
question Further information is requested

Comments

@javapythonphp
Copy link

Before Asking 在提问之前

  • I have read the README carefully. 我已经仔细阅读了 README 上的操作指引。

  • I have pulled the latest code of main branch to run again and the problem still existed. 我已经拉取了主分支上最新的代码，重新运行之后，问题仍不能解决。

Search before asking 先搜索，再提问

  • I have searched the Data-Juicer issues and found no similar questions. 我已经在 issue列表 中搜索但是没有发现类似的问题。

Question

以ray模式启动，当内存不足的时候,会溢写到磁盘吗？我在处理104G数据,内存消耗却达到了500多g

Additional 额外信息

No response
@javapythonphp javapythonphp added the question Further information is requested label Feb 11, 2025
@danielhjz
Copy link
Contributor

有解决吗，

@pan-x-c
Copy link
Collaborator

pan-x-c commented Feb 20, 2025

当前 Ray Object Store 确实支持磁盘映射，但是仅适用于存储于 Object Store 中的数据，如果是 Ray Actor 或 Task 本身超过了内存限制则无法处理，详细情况建议参考 Ray 的官方文档

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@danielhjz @pan-x-c @javapythonphp