Docker build fails with confusing ZFS error message

[diekhans]

Description

Update: This might have been triggered by being ZFS degraded, however it continued after resilver. Doing a prune -a fixed it.

Docker could do a better job of suggesting solutions in the error message rather than just passing through ZFS errors.

---

Running docker build with images on a ZFS file system that is degraded generate a confusing error:

ERROR: failed to solve: failed to prepare sha256:a790f937a6aea2600982de54a5fb995c681dd74f26968d6b74286e06839e4fb3 as i168knaqlfwod6v32797svyoh: exit status 2: "/usr/sbin/zfs snapshot data/d4cc76937f95999fda2d4dcbc4fcdca294e73d7f6202e62d62962ba9ce650287@913636948" => cannot open 'data/d4cc76937f95999fda2d4dcbc4fcdca294e73d7f6202e62d62962ba9ce650287': dataset does not exist
usage:
	snapshot [-r] [-o property=value] ... <filesystem|volume>@<snap> ...

For the property list, run: zfs set|get

For the delegated permission list, run: zfs allow|unallow

Reproduce

docker-build.out.txt
userApps.dockerfile.txt
zpool.stat.txt

Expected behavior

docker should be able to build a snapsnot even when ZFS file system is degraged.

If it can't for some reason, it should at least give a useful error message

thanks!

docker version

% docker version                                                                               
Client: Docker Engine - Community                                                                            
 Version:           27.4.1                                                                                   
 API version:       1.47                                                                                     
 Go version:        go1.22.10                                                                                
 Git commit:        b9d17ea                                                                                  
 Built:             Tue Dec 17 15:47:23 2024                                                                 
 OS/Arch:           linux/amd64                                                                              
 Context:           default                                                                                  
                                                                                                             
Server: Docker Engine - Community                                                                            
 Engine:                                                                                                     
  Version:          27.4.1                                                                                   
  API version:      1.47 (minimum version 1.24)                                                              
  Go version:       go1.22.10                                                                                
  Git commit:       c710b88                                                                                  
  Built:            Tue Dec 17 15:45:36 2024                                                                 
  OS/Arch:          linux/amd64                                                                              
  Experimental:     false                                                                                    
 containerd:                                                                                                 
  Version:          1.7.24                                                                                   
  GitCommit:        88bf19b2105c8b17560993bee28a01ddc2f97182                                                 
 runc:                                                                                                       
  Version:          1.2.2                                                                                    
  GitCommit:        v1.2.2-0-g7cb3632                                                                        
 docker-init:                                                                                                
  Version:          0.19.0                                                                                   
  GitCommit:        de40ad0

docker info

% docker info                                                                                  
Client: Docker Engine - Community                                                                            
 Version:    27.4.1                                                                                          
 Context:    default                                                                                         
 Debug Mode: false                                                                                           
 Plugins:                                                                                                    
  buildx: Docker Buildx (Docker Inc.)                                                                        
    Version:  v0.19.3                                                                                        
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx                                                  
  compose: Docker Compose (Docker Inc.)                                                                      
    Version:  v2.32.1                                                                                        
    Path:     /usr/libexec/docker/cli-plugins/docker-compose                                                 
                                                                                                             
Server:                                                                                                      
 Containers: 23                                                                                              
  Running: 0                                                                                                 
  Paused: 0                                                                                                  
  Stopped: 23                                                                                                
 Images: 99                                                                                                  
 Server Version: 27.4.1                                                                                      
 Storage Driver: zfs                                                                                         
  Zpool: data                                                                                                
  Zpool Health: DEGRADED                                                                                     
  Parent Dataset: data                                                                                       
  Space Used By Parent: 40241145674690                                                                       
  Space Available: 48629411644478                                                                            
  Parent Quota: no                                                                                           
  Compression: off                                                                                           
 Logging Driver: json-file                                                                                   
 Cgroup Driver: systemd                                                                                      
 Cgroup Version: 2                                                                                           
 Plugins:                                                                                                    
  Volume: local                                                                                              
  Network: bridge host ipvlan macvlan null overlay                                                           
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive                                                                                             
 Runtimes: io.containerd.runc.v2 runc                                                                        
 Default Runtime: runc                                                                                       
 Init Binary: docker-init                                                                                    
 containerd version: 88bf19b2105c8b17560993bee28a01ddc2f97182                                                
 runc version: v1.2.2-0-g7cb3632                                                                             
 init version: de40ad0                                                                                       
 Security Options:                                                                                           
  seccomp                                                                                                    
   Profile: builtin                                                                                          
  cgroupns                                                                                                   
 Kernel Version: 5.14.0-503.19.1.el9_5.x86_64                                                                
 Operating System: Rocky Linux 9.5 (Blue Onyx)                                                               
 OSType: linux                                                                                               
 Architecture: x86_64                                                                                        
 CPUs: 192                                                                                                   
 Total Memory: 3.937TiB                                                                                      
 Name: hgwdev                                                                                                
 ID: 102c1e67-7468-423e-8dd6-7cc2775e3a50                                                                    
 Docker Root Dir: /data/docker                                                                               
 Debug Mode: false                                                                                           
 Experimental: false                                                                                         
 Insecure Registries:                                                                                        
  127.0.0.0/8                                                                                                
 Live Restore Enabled: false

Additional Info

No response

[diekhans]

The build continues to fail with the same confusing error message now that ZFS pool has been resilvered.

We will gather more information

[diekhans]

docker system prune -a

fixes this, however a far clearly error message would be useful