mirai

[shikokuchuo]

Re. #350 for example, have you thought about using mirai instead of callr?

Happy to run through the differences, perhaps on a call.

[sebffischer]

Hi, thanks for the suggestion! I am not sure how big the advantage would be.
As long as we support some features (such as parallelization via future), we need to make sure anyway that we marshal objects at the right places, so we have to deal with the problem anyway. I think a case could be made if we could replace everything with mirai, but that's not really my call to make.
However, I could definitely see us supporting mirai as an option to encapsulate the training. Would there be any other advantages for us?

Regarding a meeting, I think other people from the mlr3 team would also need to be there and I will soon be on vacation (for 3 weeks from the 17th onward), and things are a bit hectic before that. So I could only meet in march.

Also pinging @mb706 @berndbischl @be-marc to get their input

[shikokuchuo]

Yes, that sounds like a good time to meet after you're back! I was actually referring specifically to the use of callr, which serializes to disk. I remember @dfalbel saying before that this was slow and a possible bottleneck.

In any case, it would be useful to know what the current pain points are in the mlr3 context. mirai is under active development, so even if it doesn't cover these now, it possibly could if the case is made.

[be-marc]

I see great advantages there. I think maybe with mirai, parallelization and encapsulation could become the same thing? At the moment we use future for parallelization and callr to guard the process against errors and segfaults. @shikokuchuo But mirai could handle both for us, right? Does restarting daemons after a segfaults work?

I already tested to replace future with mirai in mlr3. But I haven't thought about replacing the encalsulation with it yet. That would require a bit more work because we have to put the fallback/encapsulation handling from the worker to the main session.

I was actually referring specifically to the use of callr, which serializes to disk.

Yes I recently crashed a very large cluster because the callr encapsulation overloaded the file system. So that would also be a big advantage.

I would be happy to meet with you to discuss this further. But we can also collect more ideas here.

[shikokuchuo]

Thanks for these - let's meet once Seb is back. But feel free to post more ideas/observations in the meantime.

To your specific questions - yes it's easy to restart daemons with the launch functions. Also from your description, I think mirai by default (with dispatcher) does indeed do both jobs - for example an OOM would normally cause the dispatcher process to get killed rather than the main process.