kubeconfig path is ignored in buildx configuration

[kwohlfahrt]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

I have a buildx kubernetes builder, with the kubeconfig parameter set. However, it still seems to use the default kubeconfig (~/.kube/config) instead of the value of the parameter.

Expected behaviour

I create a buildx builder, with the KUBECONFIG environment variable set. This creates the builder, with the kubeconfig= parameter set to the value of the KUBECONFIG environment variable:

KUBECONFIG=$HOME/.docker/kubeconfig docker buildx create \
  --name kube --driver kubernetes --platform linux/amd64 --node buildx-amd64 \
  --driver-opt nodeselector="kubernetes.io/arch=amd64",rootless=true,namespace=buildx

--bootstrap is deliberately omitted, since the builder is already deployed to the kubernetes cluster. Then, when I do docker buildx ls, the builder should be listed as running:

kube *          kubernetes                                                                                                   
  buildx-amd64  kubernetes:///kube?deployment=buildx-amd64&kubeconfig=%2FUsers%2Fkai%2F.docker%2Fkubeconfig running v0.11.3  linux/amd64*, linux/amd64/v2, linux/amd64/v3, linux/386

Actual behaviour

If I run docker buildx ls, the builder is listed as inactive, even though it includes the kubeconfig= parameter.

> docker buildx ls
kube *          kubernetes                                                                                                    
  buildx-amd64  kubernetes:///kube?deployment=buildx-amd64&kubeconfig=%2FUsers%2Fkai%2F.docker%2Fkubeconfig inactive          linux/amd64*

If I run the same command with the KUBECONFIG env variable set, it shows as running:

> env KUBECONFIG=$HOME/.docker/kubeconfig docker buildx ls
kube *          kubernetes                                                                                                   
  buildx-amd64  kubernetes:///kube?deployment=buildx-amd64&kubeconfig=%2FUsers%2Fkai%2F.docker%2Fkubeconfig running v0.11.3  linux/amd64*, linux/amd64/v2, linux/amd64/v3, linux/386

Buildx version

github.com/docker/buildx v0.10.3 79e156b

Docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.3)
  compose: Docker Compose (Docker Inc., v2.15.1)
  dev: Docker Dev Environments (Docker Inc., v0.1.0)
  extension: Manages Docker extensions (Docker Inc., v0.2.18)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.25.0)
  scout: Command line tool for Docker Scout (Docker Inc., v0.6.0)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 19
 Server Version: 20.10.23
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.49-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.765GiB
 Name: docker-desktop
 ID: LOA7:7WN7:Q7AJ:R356:DNBR:ARTM:I5WP:FTQX:NFW3:6CEC:2LBA:TYUB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE       DRIVER/ENDPOINT                                                                                     STATUS  BUILDKIT PLATFORMS
buildx          docker-container                                                                                                     
  buildx0       unix:///var/run/docker.sock                                                                         stopped          
kube *          kubernetes                                                                                                           
  buildx-amd64  kubernetes:///kube?deployment=buildx-amd64&kubeconfig=%2FUsers%2Fkai%2F.docker%2Fkubeconfig         running v0.11.3  linux/amd64*, linux/amd64/v2, linux/amd64/v3, linux/386
kube-kai        kubernetes                                                                                                           
  buildx-amd64  kubernetes:///kube-kai?deployment=buildx-amd64&kubeconfig=%2FUsers%2Fkai%2F.docker%2Fkubeconfig-kai running v0.11.3  linux/amd64*, linux/amd64/v2, linux/amd64/v3, linux/386
default         docker                                                                                                               
  default       default                                                                                             running 20.10.23 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
desktop-linux   docker                                                                                                               
  desktop-linux desktop-linux                                                                                       running 20.10.23 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Configuration

No Dockerfile was used. See above for commands.

Logs

No response

Additional info

The reason this is required, is that I want to have two different buildx builders, one in a production cluster, and one in a staging cluster.

[crazy-max]

Looking at

buildx/builder/node.go

Line 80 in 4a73abf

kcc, err = ctxkube.ConfigFromContext(n.Endpoint, contextStore)

It should already take the config from the endpoint. Any idea @AkihiroSuda?

[kwohlfahrt]

This seems to be fixed in newer versions, currently using github.com/docker/buildx v0.11.2-desktop.5 f20ec1393426619870066baba9618cf999063886

[crazy-max]

@kwohlfahrt Yes this has been fixed in 0.11.2 (#1957)