Migrating a "legacy" workflow using Make to Windmill

[jcmcken]

We have a "legacy" workflow that's implemented using Make. It has multiple components in its graph, some in Bash, some in Python. What concerns me most is the Bash parts of the workflow. These parts tend to have specific runtime dependencies (e.g. specific CLIs that need to be installed).

From what I can gather, there seem to be some holes in what Windmill supports in this regard:

For example, Kubernetes support seems to be missing. It would be nice to be able to launch a Kubernetes Job using an image of my choice, such that I can completely control the runtime environment (e.g. dependencies) -- a la Argo Workflow. Then I can run whatever routines are required and store the output as an artifact in Windmill.

I can try to use the Bash built-in runtime, but then it's unclear how to get my dependencies installed. I don't want to install them every job run ideally. I also don't really want to be modifying the Windmill workers at all, since then things aren't particularly reproducible. And running like what's described here is not that appealing for many reasons -- It's not built-in. It needs to copy-pasta'd all over the place. It needs host-level access. There's no garbage collection of pods launched this way (more cruft to add into my script).

[rubenfiszel]

Hi,

Could your use-case be filled by having any binaries or CLI being pre-installed and thus available to any bash script without ceremony:
https://www.windmill.dev/docs/advanced/preinstall_binaries

It assumes you are self-hosting, is that the case?

[jcmcken]

It's better than nothing. Although not ideal, since I think it implies either multiple pools of workers with different images, or else a big omnibus worker image.

Yes, self-hosting, but just evaluating capabilities at the moment.

[rubenfiszel]

If you go through that link, you will see that using INIT_SCRIPT you do not need to modify the base image which is the point

[jcmcken]

Right, but let's say I have a bunch of dependencies, Conda, make, custom libs, etc. 10s of GB worth of dependencies. I don't want to have to reinstall them every single time a worker pod restarts. Especially since it's unlikely to reproduce exactly the same dependencies every time.