How to deal with evolving schemas?

[o-smirnov]

As raised by @JSKenyon here #18. But of course this problem is not unique to QuartiCal or pfb-clean. Wsclean also likes to introduce new parameters with new versions. How to deal with this cleanly?

[o-smirnov]

Just nothing that there's two problems to address:

• How to propagate schema files into cult-cargo without making QuartiCal or pfb-clean an explicit dependency.

• How to provide multiple schema versions within cult-cargo. The brute-force way is to propagate version names into the cab name (so e.g. the cab name becomes quartical-0.2.2, while quartical itself refers to perhaps the newest version). A more elaborate way would be to add a versions field to the parameter definition where one could specify, somehow, which version a particular parameter is valid for.

[JSKenyon]

To be clear though, I am not quite sure if it would be possible to avoid updating cab definitions by hand for applications not using stimela as a parser. At the very least, as mentioned in my other comment, having PRs opened automatically on release/tag would at least make keeping track a little simpler.

[o-smirnov]

Yeah no avoiding doing this by hand for third-party apps, this is true. But at least we can come up with the most labour-saving way to do this for the ones we control...

[sjperkins]

I did some quick googling for yaml registry schemas, but notihing OSS and obvious popped up. I'm also certain someone else has had to think of this problem before (k8s was one of the search results).

Perhaps with more digging there wouldn't be a need to reinvent the wheel.

[o-smirnov]

But do we even need to invent this wheel? In a sense, the cult-cargo repo is the schema registry already. A new release of an app comes along, we push an update and issue an "rc" release for the brave. Seems simple enough, we just need a smoother way to copy the yamls over.

[sjperkins]

Maybe not, I know very little about cult-cargo, but it sounds like it could be repurposed.

FWIW vcpkg keeps a versions directory for different ports (cabs?). For example fftw3:

https://github.com/microsoft/vcpkg/blob/master/versions/f-/fftw3.json

These in turn refer to different git commits referencing versions of the port (cab?) in this directory

https://github.com/microsoft/vcpkg/tree/master/ports/fftwpp

[JSKenyon]

Copied from #18. I think that the correct thing to do is to have a GitHub action that handles this automatically. This could be on either the application repo or on cult-cargo. The latter would be preferable but the former is probably easier. Basically, on release, the action should pick out the required files (a reason to have this on the application repos) and automatically make a PR into cult-cargo pushing those files to some folder based on the current release version. This means that the application developer only has to worry about it once, and it simply becomes merging a PR on the cult-cargo end.

[tmolteno]

Yep that would be fine for me (https://peps.python.org/pep-0440/#version-specifiers).

[landmanbester]

It is possible to extend configuratt's _include statement to support a URL, so e.g. to pull the schema directly off the pfb-clean repo. But then this undermines reproducibility -- and requires a runtime internet connection, which clusters don't always grant you. Still might be a handy option to have for the bleeding-edge people. But a "reproducible" recipe must, by necessity, come with a fixed snapshot of the schemas (and docker images) used by that recipe. A "specific release of cult-cargo" provides exactly that snapshot -- but maybe there's an even easier way to do it which we just haven't figured out yet.

I'm wondering if it would be possible to use such a mechanism to make the dev version of cult-cargo a bit more dev friendly. For example, could we not set things up so that the dev version of cult-cargo fetches config files from the application repo when using the latest form of cab specification? latest would still refer to a specific version of a cab for any cult-cargo release (something that can probably be frozen in using github actions or the freeze mechanism @JSKenyon is proposing) so recipes relying on official releases would still be reproducible but we don't guarantee reproducibility when using the the dev version of cult-cargo. This way there is only one source of truth for the config and this automatically gets propagated to cult-cargo upon release (effectively getting added to the registry) and users are also able to quickly test fixes simply by installing a dev version of cult-cargo without the need to install other packages. Of course, they would need to set their image-pull-policies to always to get up to date Docker images but this is still something that happens at the level of the config without the need for making new releases. Just thinking out loud here, not sure if this is actually feasible/practical. Thoughts?

[o-smirnov]

Maybe the dev version of cult-cargo should be a separate package called fresh-cargo -- no reason having one package that tries to be all things to all people -- and then it can have the fancy runtime config schema fetch shizzle?

[landmanbester]

hip-cargo 😎

We could probably also have main versions for this, that way latest keeps its current meaning

[tmolteno]

I like the idea of explicit versioning

• cult-cargo:testing (refer to the upcoming next stable, this is the version that will become the next release)
• cult-cargo:2024.02 (refer to an explicit release)
• cult-cargo without versioning should throw an error

Using a different package called fresh-cargo: love it, and makes things easier (as an alternative to the first option above). Another name might be unstable-cargo to hint to the user the gravity of using this in production scripts.