BUILDX_EXPERIMENTAL option breaks registry caching for remote builds

[ehlertjd]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

When building with the remote driver, the --cache-from type=registry option appears to be ignored, as long as BUILDX_EXPERIMENTAL=1.

Expected behaviour

The remote cache should be used to skip building any layers that have not been modified.

Actual behaviour

Each layer is rebuilt rather than reading from remote cache.
Strangely, the --cache-to option seems to be honored, as the build layers are written to the remote registry.

Buildx version

github.com/docker/buildx v0.22.0 18ccba0

Docker info

Client: Docker Engine - Community
 Version:    28.0.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.22.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.34.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 28.0.4
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-1021-azure
 Operating System: Ubuntu 24.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.61GiB
 Name: vm-justindev
 ID: af9b8f7b-71ea-4de8-bf90-f957769e3495
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: justinehlert
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE             DRIVER/ENDPOINT            STATUS    BUILDKIT   PLATFORMS
remote-builder*       remote
 \_ remote-builder0    \_ tcp://localhost:1234   running   v0.20.2    linux/amd64 (+4), linux/386
ctx                   docker
 \_ ctx                \_ ctx                    running   v0.20.2    linux/amd64 (+4)
default               docker
 \_ default            \_ default                running   v0.20.2    linux/amd64 (+4)

Configuration

Dockerfile

FROM alpine:3.21.3

RUN apk add --no-cache bash

RUN printf "#!/usr/bin/env bash\necho Hello World" >> hello.sh
RUN chmod +x hello.sh

ENTRYPOINT ["/bin/bash", "hello.sh"]

Script to reproduce the issue:
build.sh

#!/bin/bash
set -e

cache_repo=<cache repository>

if [[ "$1" == "experimental" ]]; then
  export BUILDX_EXPERIMENTAL=1
  echo "BUILDX_EXPERIMENTAL=1 enabled"
else
  export BUILDX_EXPERIMENTAL=0
fi

docker run -d --name buildkitd \
  --privileged \
  -p 1234:1234 \
  moby/buildkit:latest \
  --addr tcp://0.0.0.0:1234

if ! grep -q remote-builder <<<"$(docker buildx ls)"; then
    docker buildx create --use \
            --name=remote-builder --driver=remote \
            tcp://localhost:1234
fi

docker buildx build . --builder=remote-builder --progress=plain \
        --cache-to type=registry,ref=$cache_repo,mode=max \
        --cache-from type=registry,ref=$cache_repo

docker rm -f buildkitd
docker buildx rm remote-builder

Reproduction steps:

1. Tested on Ubuntu 24.04 with Docker 28.0.4 and containerd-snapshotter enabled
2. Update cache_repo in build.sh
3. Run build.sh without any arguments - first time will populate the registry cache, subsequent builds will use the cache
4. Run build.sh experimental - Observe that the registry cache is not used

Build logs

#0 building with "remote-builder" instance using remote driver

#1 [internal] connecting to local controller
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile:
#2 transferring dockerfile: 211B done
#2 DONE 0.1s

#3 [internal] load metadata for docker.io/library/alpine:3.21.3
#3 ...

#4 [auth] library/alpine:pull token for registry-1.docker.io
#4 DONE 0.0s

#3 [internal] load metadata for docker.io/library/alpine:3.21.3
#3 DONE 0.8s

#5 [internal] load .dockerignore
#5 transferring context: 2B done
#5 DONE 0.1s

#6 [1/4] FROM docker.io/library/alpine:3.21.3@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c
#6 resolve docker.io/library/alpine:3.21.3@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c 0.1s done
#6 sha256:f18232174bc91741fdf3da96d85011092101a032a93a388b79e99e69c2d5c870 3.64MB / 3.64MB 0.1s done
#6 extracting sha256:f18232174bc91741fdf3da96d85011092101a032a93a388b79e99e69c2d5c870 0.1s done
#6 DONE 0.4s

#7 [2/4] RUN apk add --no-cache bash
#7 0.094 fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/main/x86_64/APKINDEX.tar.gz
#7 0.305 fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/community/x86_64/APKINDEX.tar.gz
#7 0.571 (1/4) Installing ncurses-terminfo-base (6.5_p20241006-r3)
#7 0.586 (2/4) Installing libncursesw (6.5_p20241006-r3)
#7 0.601 (3/4) Installing readline (8.2.13-r0)
#7 0.616 (4/4) Installing bash (5.2.37-r0)
#7 0.638 Executing bash-5.2.37-r0.post-install
#7 0.641 Executing busybox-1.37.0-r12.trigger
#7 0.644 OK: 9 MiB in 19 packages
#7 DONE 1.0s

#8 [3/4] RUN printf "#!/usr/bin/env bash\necho Hello World" >> hello.sh
#8 DONE 0.2s

#9 [4/4] RUN chmod +x hello.sh
#9 DONE 0.2s

#10 exporting cache to registry
#10 preparing build cache for export
#10 writing layer sha256:3e217c93b02ce141b8f05b2b5607b75ae219731986b7a7f41762b90f19efcd85
#10 ...

#11 [auth] justinehlert/test:pull,push token for registry-1.docker.io
#11 DONE 0.0s

#10 exporting cache to registry
#10 writing layer sha256:3e217c93b02ce141b8f05b2b5607b75ae219731986b7a7f41762b90f19efcd85 0.9s done
#10 writing layer sha256:cf451abab2ee909ad8d233c23455c9e5a7bff8044472ecf0112c677d223dbe22
#10 writing layer sha256:cf451abab2ee909ad8d233c23455c9e5a7bff8044472ecf0112c677d223dbe22 0.7s done
#10 writing layer sha256:f18232174bc91741fdf3da96d85011092101a032a93a388b79e99e69c2d5c870
#10 writing layer sha256:f18232174bc91741fdf3da96d85011092101a032a93a388b79e99e69c2d5c870 0.2s done
#10 writing layer sha256:f3529faa984079cb407ff62418c9bb542ab304b50e58c04330f7e8cb7ee72a0e
#10 writing layer sha256:f3529faa984079cb407ff62418c9bb542ab304b50e58c04330f7e8cb7ee72a0e 0.9s done
#10 writing config sha256:fc2ad7a73ab26a0eb76252eee7bc34851d91e3516a5a81ae84753bc388fd8223
#10 writing config sha256:fc2ad7a73ab26a0eb76252eee7bc34851d91e3516a5a81ae84753bc388fd8223 0.7s done
#10 writing cache manifest sha256:53f0aa2e6e303175a386c0618ee72c7553de6bc1f278a4b0d808b866411cfaf2
#10 preparing build cache for export 5.0s done
#10 writing cache manifest sha256:53f0aa2e6e303175a386c0618ee72c7553de6bc1f278a4b0d808b866411cfaf2 0.9s done
#10 DONE 5.0s
WARNING: No output specified with remote driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

Additional info

No response

[naterichman]

I did some debugging on both buildx and buildkit. When running with BUILDX_EXPERIMENTAL=true there are never any cacheImports received by the loadResult function whereas with BUILDX_EXPERIMENTAL=false the cacheImports are populated.

I'm not exactly sure where the difference is, but I traced the code-paths down to BUILDX_EXPERIMENTAL=true using the NewResultHandle branch here and BUILDX_EXPERIMENTAL=false using the plain c.Build branch.

I wasn't able to find any concrete difference between the two branches though

[tonistiigi]

This should be resolved by #3124 I think

[tonistiigi]

@ehlertjd Can you retry not what we have merged #3126 ?