Launch a bunch of long running processes, bring them down easily

[lucasw]

I've looked at the documentation some and see that indefinitely running processes aren't the main goal, but it can be done. Can the jobs make heavy use of networking to talk to each other and other computers?

I can describe my use case more here but I have this already:

https://users.rust-lang.org/t/cli-tool-written-in-rust-to-launch-a-bunch-of-processes-given-a-config-file/120630

[bobbobbio]

Hi, thanks for your interest.

You are correct that indefinitely running processes aren't the main goal, but technically it can allow it. Maybe the main issue you will run up against is that currently there isn't great tools for streaming back the output (although there is a remote TTY feature that could maybe be used for such a thing)

As for networking though, we don't currently support jobs accessing the host's networking unless the job is run "locally" (not sent to a remote worker.) Some kind of networking communication between jobs is a planned feature at some point though.

[lucasw]

Not getting the stdout isn't ideal though for most of the processes I run I can send out the same information over the network.

Running locally is fine in order to get full network access.

Is it possible to run a job without a container, run it in the outer filesystem and with user permissions? I want everything in containers eventually but for prototyping rapidly I'd rather not have to do that extra setup, or have that additional layer of separation when debugging.

For now I'm going to try something very minimal with tokio process command and have the basics working here: https://github.com/lucasw/tf_roslibrust/blob/roslibrust/launcher/src/main.rs which uses toml config files like this. I'll likely outgrow that though, and look into maelstrom again (it'd be great to hear more about it at something like one of these meetups https://www.meetup.com/join-srug/, I notice from your profile you're in the area).

[bobbobbio]

Is it possible to run a job without a container, run it in the outer filesystem and with user permissions? I want everything in containers eventually but for prototyping rapidly I'd rather not have to do that extra setup, or have that additional layer of separation when debugging.

The jobs do run with user permissions, maelstrom doesn't need root access at all. There is no way to turn of the containerization.

it'd be great to hear more about it at something like one of these meetups https://www.meetup.com/join-srug/, I notice from your profile you're in the area.

yeah that's a good idea