Adopting Zstandard/"zstd" by default

[tianon]

I'm filing this as a tracking issue for switching to Zstandard/"zstd" compression of layer blobs in Official Images so we can have a centralized place to point folks and to collect/dump relevant information.

To be very explicit, I am personally of the opinion that we would gain far more as an ecosystem by switching to canonically uncompressed blobs instead (and thus relying on at-rest and client-/server-negotiated transport compression, as suggested in opencontainers/distribution-spec#235 and linked issues/PRs such as containerd/containerd#8166 and distribution/distribution#3900), as this solves more interesting problems than simply swapping out gzip for zstd, but I do not think it's appropriate to block zstd adoption here entirely based on that opinion (especially since that idea gaining traction would just as easily mean dropping zstd as it does dropping gzip).

In order to make the determination as to whether we're past the "adoption curve" sufficiently to feel good about changing our defaults (ie, to shrink the list of stragglers who will come and complain about their outdated software no longer working with our images), I am collecting here a list of popular runtimes/tools, and the version/date that they added zstd support.

Runtime/Tool	Versions	Dates	Repology
Docker/Moby	23+	2023-02-01	https://repology.org/project/docker/badges
containerd	1.5+	2021-05-03	https://repology.org/project/containerd/badges
crane/go-containerregistry	0.13+	2023-01-24
containers/image (library)	4+	2019-10-01
podman	1.6.2+	2019-10-19	https://repology.org/project/podman/badges
buildah	1.11.3+	2019-10-04	https://repology.org/project/buildah/badges
skopeo	0.1.40+	2019-10-29	https://repology.org/project/skopeo/badges
cri-o	1.17+	2020-02-07	https://repology.org/project/cri-o/badges
Singularity	3.6+ ?	2020-07-14	https://repology.org/project/singularity-container/badges

Additionally, I am compiling a companion table of distribution releases and their relevant EOL dates in cases where they have runtimes/tools that are not recent enough to have zstd support.

Runtime/Tool	Distro	Version	EOL Date
Docker	Debian Trixie (13)	26+
	Bookworm (12)	20.10	2028-06-30
	Bullseye (11)	20.10	2026-08-31
	Buster (10)	18.09	2024-06-30
	Ubuntu Noble (24.04)	24+	2029-06
	Jammy (22.04)	24+	2027-06
	Focal (20.04)	24+	2025-04
containerd	Debian Trixie (13)	1.7+
	Bookworm (12)	1.6
	Bullseye (11)	1.4
	Buster (10)	-
	Ubuntu Noble (24.04)	1.6
	Jammy (22.04)	1.6
	Focal (20.04)	1.6

(The more I compile this table, the less happy with it I am 😭 I can't figure out a clean way to organize it that still includes all the data I need it to, but isn't so heinously verbose or superwide.)

See also:

• https://twitter.com/cpuguy83/status/1778083160657662198
• https://twitter.com/tianon/status/1778116752939032996

[tianon]

To reiterate the conclusion I've just posted on moby/moby#48328 (comment) here (with the DOI slant), I think those Debian Stable and Oldstable (Bookworm and Bullseye) versions still being stuck on 20.10 (and thus having no zstd support) means it's going to be several more years yet before DOI can reasonably consider switching to zstd -- at the very least, we'd need the release of Debian Trixie (which has a new enough version), but ideally a bit further into the Trixie support window before we consider dropping support for Bookworm users (maybe even all the way to that 2028 date, but at the very least likely the 2026 date of Bullseye's EOL 😭).

Perhaps this is useful context for why I think uncompressed canonical layers is a better solution for the industry overall -- then registries and runtimes can decide which compression they want to use for storage (making whatever appropriate speed/space/CPU tradeoffs for their own situation), and can coordinate at the transport level which compression to use over-the-wire based on what they're willing to support. Current users would all be mostly unaffected because the "failure mode" is simply that layers are bigger than they used to be, but everything still works fine otherwise (so now upgrading is a very compelling carrot vs a stick/broken workflows). That would also resolve some of the interesting struggles around layer IDs vs "diff IDs" that the image specification has today. Doing "random access" from the registry API would then be trivial range requests across the tar (needing only a map of tar entries to offsets vs complex compression coordination). 🤷

(To put all of this another way, I don't think zstd solves enough "interesting" problems to be worth breaking so many users at the present time. ❤️)

[cgwalters]

Perhaps this is useful context for why I think uncompressed canonical layers is a better solution for the industry overall

Yes, I agree overall. When I made ostree it was very intentionally inspired by git, which does not checksum compressed content and so neither have this problem.

(tangential) One thing related to this is I've been having a vague thought that we could try to enourage signing tools (which commonly sign the manifest today) to also have a secondary signature that covers only the uncompressed content that can also be carried around (I think it'd just be the manifest without the compressed checksum layer IDs?...gets into issues around canonical json) etc. if we need to substitute values reproducibly).

[jonjohnsonjr]

I know this has been discussed to death in other (probably more appropriate) contexts but I made a diagram and wanted to post it somewhere public before I lost it due to Chrome crashing and being unable to recover my tabs for inexplicable reasons.

Anyway, I think adopting zstd is a good idea, but it would be swell if it was the seekable format rather than just "normal" zstd. By default, zstd allows for effectively infinite window sizes, which make soci-style tricks for making a gzip archive seekable impossible. (Aside: IETF is dealing with this by suggesting that windows be kept to 8MB for HTTP.) The zstd seekable format splits the archive into multiple frames (kind of like stargz) and embeds a seek table for skipping ahead to those frames.

There's also zstd:chunked which does a similar thing to the zstd seekable format but also embeds an index of tar metadata (similar to how stargz works).

I do not like stargz or zstd:chunked because they couple the make-the-compressed-archive-seekable solution to tar format. We can solve the seekability problem separate from the problem of having a tar index, which would mean we could reuse the same tar index across different compression formats, and we could reuse the compression seekability metadata for non-tar artifacts (i.e. OCI artifacts).

Something we discussed in OCI land a while back was doing compression-in-transit and using the uncompressed content as the in-registry form. This solves a bunch of problems for consumers of bits but forces some complexity and/or cost onto the registry. I still think doing this would be the right trade-off, but I'm no expert.

I promised a diagram:

Uncompressed blobs can consume the tar index directly and don't need any special handling to seek.

The seekable zstd format solves the "unwrapping" of zstd.

You can use soci-style external checkpointing (see here) to make gzip seekable.

With this decomposition, we can still use the gzip checkpoints or zstd seek table for non-tar contents like sqlite databases or something, whereas stargz and zstd:chunked are tightly coupled to tar contents only.

[cgwalters]

There's also zstd:chunked which does a similar thing to the zstd seekable format but also embeds an index of tar metadata (similar to how stargz works).

Yeah, it's even more complicated than that unfortunately because e.g. there's also a third copy of the tar headers in tar-split style to ease reconstructing the image...I was arguing around that space that we really want is a "canonical tar" instead since obviously in this model we control the tar stream and don't need to care about reconstructing from an arbitrary tarball produced by someone else.

I do not like stargz or zstd:chunked because they couple the make-the-compressed-archive-seekable solution to tar format.

Hmm this issue doesn't seem the right place to debate this though. Shouldn't we take this to opencontainers/distribution-spec#235 ?

In particular what I'd say here is I have a sense that if we're breaking compatibility on the compression side, we may as well also actually look at a partial break on the tar side too, adding an index directly there say.

[giuseppe]

In particular what I'd say here is I have a sense that if we're breaking compatibility on the compression side, we may as well also actually look at a partial break on the tar side too, adding an index directly there say.

stargz adds the index as part of the tar stream.

Instead, zstd:chunked uses the compression layer to hide the index. The information is added through skippable frames so it doesn't "break" compatibility as clients that do not understand the additional metadata will simply ignore it as required by the zstd format: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#skippable-frames.