Windows node with ContainerD can't run flannel and kubeproxy daemonsets

[vitaliy-leschenko]

Describe the bug
Flannel and kube-proxy pods do not start.

To Reproduce

1. setup on-premise cluster (v1.19.0) with flannel (vxlan)

• flannel image: vleschenko/flannel:0.13.0 (it is support WS2004)
• kubeproxy image: vleschenko/kube-proxy:v1.19.0 (it is support WS2004)

2. setup Windows Server Standard 2004
3. install containerd
4. prepare node
5. kubeadm join

Node successfully joined to cluster:

PS C:\Users\v.leschenko> kubectl get nodes -owide
NAME           STATUS   ROLES    AGE    VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                  KERNEL-VERSION       CONTAINER-RUNTIME
k8s            Ready    master   241d   v1.19.0   192.168.2.20   <none>        Ubuntu 18.04.5 LTS        4.15.0-123-generic   docker://19.3.12
k8s-us1804-a   Ready    <none>   241d   v1.19.0   192.168.2.25   <none>        Ubuntu 18.04.5 LTS        4.15.0-123-generic   docker://19.3.12
k8s-ws2004-a   Ready    <none>   47m    v1.19.0   192.168.2.30   <none>        Windows Server Standard   10.0.19041.630       containerd://1.4.1

Pods stuck on ContainerCreating

kube-flannel-ds-windows-amd64-zclk9   0/1     ContainerCreating   0          18m     192.168.2.30   k8s-ws2004-a   <none>           <none>
kube-proxy-windows-gqjnh              0/1     ContainerCreating   0          18m     <none>         k8s-ws2004-a   <none>           <none>

Error message:

  Warning  FailedCreatePodSandBox  4m22s (x63 over 17m)  kubelet, k8s-ws2004-a  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fe980b773114549b7144eaa3bbd44f277394ad7be669697a442141372da3f29c": error creating endpoint hcnCreateEndpoint failed in Win32: IP address is either invalid or not part of any configured subnet(s). (0x803b001e) {"Success":false,"Error":"IP address is either invalid or not part of any configured subnet(s). ","ErrorCode":2151350302} : endpoint config &{ fe980b773114549b7144eaa3bbd44f277394ad7be669697a442141372da3f29c_nat 8f77950d-c85e-4b74-b9dc-61ce5e671d3a  [] [{ 0}] { [.] [127.0.0.1] []} [{192.168.2.30 0.0.0.0/0 0}]  0 {2 0}}

Expected behavior
flannel and kube-proxy started

Kubernetes:

• Windows Server version: Windows Server 2004
• Kubernetes Version: 1.19.0
• CNI: ContainerD/1.4.1

[vitaliy-leschenko]

@marosset could you help with it?

[vitaliy-leschenko]

After VM reboot in kubelet logs I can see:

E1127 10:50:46.560757    3224 remote_runtime.go:113] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to setup network for sandbox "976a7614895c88721d7e5e45d598643dc90f4bd3fc9ad8fb6184374f2b4dfde1": hcnCreateNetwork failed in Win32: The object already exists. (0x1392) {"Success":false,"Error":"The object already exists. ","ErrorCode":2147947410}
E1127 10:50:46.560757    3224 kuberuntime_sandbox.go:69] CreatePodSandbox for pod "kube-flannel-ds-windows-amd64-zclk9_kube-system(d168e243-716c-47b4-8ae5-a5d01399ac1c)" failed: rpc error: code = Unknown desc = failed to setup network for sandbox "976a7614895c88721d7e5e45d598643dc90f4bd3fc9ad8fb6184374f2b4dfde1": hcnCreateNetwork failed in Win32: The object already exists. (0x1392) {"Success":false,"Error":"The object already exists. ","ErrorCode":2147947410}
E1127 10:50:46.560757    3224 kuberuntime_manager.go:730] createPodSandbox for pod "kube-flannel-ds-windows-amd64-zclk9_kube-system(d168e243-716c-47b4-8ae5-a5d01399ac1c)" failed: rpc error: code = Unknown desc = failed to setup network for sandbox "976a7614895c88721d7e5e45d598643dc90f4bd3fc9ad8fb6184374f2b4dfde1": hcnCreateNetwork failed in Win32: The object already exists. (0x1392) {"Success":false,"Error":"The object already exists. ","ErrorCode":2147947410}
E1127 10:50:46.560757    3224 pod_workers.go:191] Error syncing pod d168e243-716c-47b4-8ae5-a5d01399ac1c ("kube-flannel-ds-windows-amd64-zclk9_kube-system(d168e243-716c-47b4-8ae5-a5d01399ac1c)"), skipping: failed to "CreatePodSandbox" for "kube-flannel-ds-windows-amd64-zclk9_kube-system(d168e243-716c-47b4-8ae5-a5d01399ac1c)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-flannel-ds-windows-amd64-zclk9_kube-system(d168e243-716c-47b4-8ae5-a5d01399ac1c)\" failed: rpc error: code = Unknown desc = failed to setup network for sandbox \"976a7614895c88721d7e5e45d598643dc90f4bd3fc9ad8fb6184374f2b4dfde1\": hcnCreateNetwork failed in Win32: The object already exists. (0x1392) {\"Success\":false,\"Error\":\"The object already exists. \",\"ErrorCode\":2147947410}"

ipconfig:

Windows IP Configuration


Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . : vitaliy.org
   IPv6 Address. . . . . . . . . . . : 2a03:e2c0:1801:ff00:2551:7e41:8ae0:4fc6
   IPv6 Address. . . . . . . . . . . : 2a03:e2c0:1801:ff00:8c2c:2f9:601a:d106
   Link-local IPv6 Address . . . . . : fe80::8c2c:2f9:601a:d106%7
   IPv4 Address. . . . . . . . . . . : 192.168.2.30
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : fe80::215:5dff:fe02:903%7
                                       192.168.2.254

[marosset]

I'll take a look

[vitaliy-leschenko]

@marosset I found reason why pod doesn't start: After node restart nat network has been removed.
So, I modified StartKubelet.ps1 to recreate it like for docker. Please see this script from PR #132

Now pod starts but use wrong network: nat instead of flannel

k8s-ws1809-[a-c] - docker nodes
k8s-ws1809-d - containerd node.

You can see that containerd pods use nat 172.27.0.0/16 network instead flannel of 10.244.0.0/16.
192.168.2.0/24 - my local network

[marosset]

@vitaliy-leschenko I wonder if there is a race between when kube-proxy and flannel-ds pod is able to start the flannel service.
On docker machines I observed that flannel would remove NoSchedule taints on machines to allow for other pods like kube-proxy to be scheduled.
I don't remember seeing seeing this behavior on nodes running contianerd.

Most of my experience with containerd is with setting up CNI during node configuration time, not with having CNI plugins being deployed with via pods.

@daschott do you have any insight here?

[vitaliy-leschenko]

flannel pod has the same behaviour for docker and cotnainerd nodes.

[marosset]

If kube-proxy pod is rescheduled after flannel is started does it join the correct (flannel) network?
I'm setting up a new cluster to verify this myself.

I think I remember docker nodes getting joined to the cluster in a NotReady state then the flannel service would update the node's status setting it to Ready but cointainerd nodes would get joined in the Ready state.

[vitaliy-leschenko]

yep. even if kube-proxy (or any other pods) started after flannel it also starts with NAT ip address.

[marosset]

I'm able to repro this.
Still looking at the issue tho...

[marosset]

Sorry for the slow responses here.
I've been digging into this and have a lot more context but do not yet have a solution.

When hostNetwork is true kubelet/dockershim adds the containers to an existing network named host and skips CNI specific config (created by

sig-windows-tools/kubeadm/scripts/PrepareNode.ps1

Line 77 in 9aa36e4

docker network create -d nat host

)

It looks like containerd for Windows does not have the same behavior
https://github.com/containerd/containerd/blob/88f089354009d3df6d5556d59ce4ce2ac0717106/pkg/cri/server/sandbox_run.go#L111-L117

I also ran some testing and locally where I configured flannel on the node before joining the node to a cluster (installed / ran flanneld and added a flannel CNI config in /etc/cni/net.d) and then did not schedule a Windows flannel ds to the node and everything worked as expected.

I think some possible solutions are to

• Not bring in CNI config/plugins via a deamonset pod (install create seperate scripts that should be run after PrepareNode.ps1)
• Update contiainerd to better support hostNetwork mode for Windows.

All of the upstream tests we run against kubernetes/kubernetes install CNI as part of node preparation (for both Azure and GKE) and unfortunately these tutorials missed this behavior.

Describe the bug
Flannel and kube-proxy pods do not start.

To Reproduce

1. setup on-premise cluster (v1.19.0) with flannel (vxlan)

• flannel image: vleschenko/flannel:0.13.0 (it is support WS2004)
• kubeproxy image: vleschenko/kube-proxy:v1.19.0 (it is support WS2004)

1. setup Windows Server Standard 2004
2. install containerd
3. prepare node
4. kubeadm join

Node successfully joined to cluster:

PS C:\Users\v.leschenko> kubectl get nodes -owide
NAME           STATUS   ROLES    AGE    VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                  KERNEL-VERSION       CONTAINER-RUNTIME
k8s            Ready    master   241d   v1.19.0   192.168.2.20   <none>        Ubuntu 18.04.5 LTS        4.15.0-123-generic   docker://19.3.12
k8s-us1804-a   Ready    <none>   241d   v1.19.0   192.168.2.25   <none>        Ubuntu 18.04.5 LTS        4.15.0-123-generic   docker://19.3.12
k8s-ws2004-a   Ready    <none>   47m    v1.19.0   192.168.2.30   <none>        Windows Server Standard   10.0.19041.630       containerd://1.4.1

Pods stuck on ContainerCreating

kube-flannel-ds-windows-amd64-zclk9   0/1     ContainerCreating   0          18m     192.168.2.30   k8s-ws2004-a   <none>           <none>
kube-proxy-windows-gqjnh              0/1     ContainerCreating   0          18m     <none>         k8s-ws2004-a   <none>           <none>

Error message:

  Warning  FailedCreatePodSandBox  4m22s (x63 over 17m)  kubelet, k8s-ws2004-a  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fe980b773114549b7144eaa3bbd44f277394ad7be669697a442141372da3f29c": error creating endpoint hcnCreateEndpoint failed in Win32: IP address is either invalid or not part of any configured subnet(s). (0x803b001e) {"Success":false,"Error":"IP address is either invalid or not part of any configured subnet(s). ","ErrorCode":2151350302} : endpoint config &{ fe980b773114549b7144eaa3bbd44f277394ad7be669697a442141372da3f29c_nat 8f77950d-c85e-4b74-b9dc-61ce5e671d3a  [] [{ 0}] { [.] [127.0.0.1] []} [{192.168.2.30 0.0.0.0/0 0}]  0 {2 0}}

Expected behavior
flannel and kube-proxy started

Kubernetes:

• Windows Server version: Windows Server 2004
• Kubernetes Version: 1.19.0
• CNI: ContainerD/1.4.1

Have you solved it, how did you solve it

[sl844]

Hi, I am running into a similar issue. Any updates?

[jsturtevant]

The current way around this is to install your CNI of choice on the node after the prepare script, then you can run kubeproxy as a deamonset.

We are looking into Windows privileged container support (which comes with host network support). This is a topic for the next sig-windows meeting if you want to join and discuss further: https://docs.google.com/document/d/1Tjxzjjuy4SQsFSUVXZbvqVb64hjNAG5CQX8bK7Yda9w/edit#heading=h.3f5dhus4q8i2

[sl844]

okay. Thanks I will try that route.

[jaderoliver]

@vitaliy-leschenko Hi dude, I am have this same problem now with my win server 2019, after restart the machine they report the same error with you, could you show where you change the StartKubelet.ps1, in my case I am using just the containerd without docker you know because k8s will not suporte docker.