Help troubleshooting a cluster deployed using kubeadm

[marco74]

Hello,

on the site I read
You should now be able to curl the nginx Service on <CLUSTER-IP>:<PORT> from any node in your cluster.

After following the instructions I can curl my nginx service only on that nodes where a pod is running. On the others I got
curl: (7) Failed to connect to 10.107.64.40 port 80: No route to host
Is that intentionally or am I missing something?

Also the DNS part does not work at all for me. After following the instructions according to the page, I only get

Server:    10.96.0.10
Address 1: 10.96.0.10

nslookup: can't resolve 'my-nginx'

Since this takes that long I assume it's a timeout error.

I'm using kubernetes 1.24 on Debian Bullseye with cri-dockerd installed as mentioned on https://github.com/Mirantis/cri-dockerd

Marco

[k8s-ci-robot]

@marco74: This issue is currently awaiting triage.

SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Kartik494]

@marco74 could you provide the task reference you are facing issue on?

[marco74]

Actually I do not understand your question. I already wrote that I followed the instructions on

I followed every single step on https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/

The first problem I see is that two pods have the same IP. With that they wont be able to cummunicate with each other, so what am I missing to not get the same IP on both pods (on different nodes)?

marco@kn1:~/kubernetes/test1$ kubectl apply -f ./run-my-nginx.yaml
deployment.apps/my-nginx created
marco@kn1:~/kubernetes/test1$ kubectl get pods -l run=my-nginx -o wide
NAME                       READY   STATUS              RESTARTS   AGE   IP       NODE   NOMINATED NODE   READINESS GATES
my-nginx-df7bbf6f5-lsp2v   0/1     ContainerCreating   0          8s    <none>   kn6    <none>           <none>
my-nginx-df7bbf6f5-zkhvp   0/1     ContainerCreating   0          8s    <none>   kn3    <none>           <none>
marco@kn1:~/kubernetes/test1$ kubectl get pods -l run=my-nginx -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE   NOMINATED NODE   READINESS GATES
my-nginx-df7bbf6f5-lsp2v   1/1     Running   0          52s   172.17.0.2   kn6    <none>           <none>
my-nginx-df7bbf6f5-zkhvp   1/1     Running   0          52s   172.17.0.2   kn3    <none>           <none>
marco@kn1:~/kubernetes/test1$ kubectl get pods -l run=my-nginx -o yaml | grep podIP
    podIP: 172.17.0.2
    podIPs:
    podIP: 172.17.0.2
    podIPs:

[Kartik494]

@marco74 Could you please share which cni plugin you are using ? i have checked in my env , pod's are getting different IP's .

kubectl get pods -l run=my-nginx -o wide
NAME READY STATUS RESTARTS AGE IP NODE
my-nginx-5b56ccd65f-9b55t 1/1 Running 0 70s 192.168.212.16 workernode1
my-nginx-5b56ccd65f-x9rnz 1/1 Running 0 70s 192.168.212.15 workernode1

[marco74]

@Kartik494 Thank you for your answer. Different IPs must be necessary, otherwise one pod could not connect to the other because routing won't work, when the same subnets are used. Could you please drop me a thorough instruction set how to set up a cluster, starting right after OS-setup, then I could compare to find the mistake.

Usually I startup with a fresh installed Debian Bullseye on a Virtual-Box machine that's installed by a remastered netinst-image already including Docker installation.

common instruction set (for master and worker):

## install cri-dockerd
sudo apt-get -y install golang
# From: https://github.com/Mirantis/cri-dockerd
git clone https://github.com/Mirantis/cri-dockerd.git
cd cri-dockerd/
mkdir bin
cd src/
go get
go build -o ../bin/cri-dockerd
cd ..
sudo mkdir -p /usr/local/bin
sudo install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd
sudo cp -a packaging/systemd/* /etc/systemd/system
sudo sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
cd ..

sudo systemctl daemon-reload
sudo systemctl enable cri-docker.service
sudo systemctl enable --now cri-docker.socket

## install kubernetes
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

sudo kubeadm --cri-socket unix:///run/cri-dockerd.sock config images pull

then on master

I initialize with:

sudo kubeadm --cri-socket unix:///run/cri-dockerd.sock init

## configure in personal directory
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Then I either do

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

or

kubectl create -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml

on the worker node I run the commend that the master shows me when I hit

echo "sudo `kubeadm token create --print-join-command --ttl 30m` --cri-socket=unix:///run/cri-dockerd.sock"

back on the master I can see all my workers

marco@kn1:~$ kubectl get nodes
NAME   STATUS   ROLES           AGE   VERSION
kn1    Ready    control-plane   94s   v1.24.0
kn2    Ready    <none>          30s   v1.24.0
kn3    Ready    <none>          25s   v1.24.0
kn4    Ready    <none>          21s   v1.24.0
kn5    Ready    <none>          15s   v1.24.0
kn6    Ready    <none>          11s   v1.24.0

Then I continue with https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/

marco@kn1:~$ vim run-my-nginx.yaml
marco@kn1:~$ kubectl apply -f ./run-my-nginx.yaml
kubectl get pods -l run=my-nginx -o wide
deployment.apps/my-nginx created
NAME                       READY   STATUS    RESTARTS   AGE   IP       NODE   NOMINATED NODE   READINESS GATES
my-nginx-df7bbf6f5-bxh54   0/1     Pending   0          0s    <none>   kn4    <none>           <none>
my-nginx-df7bbf6f5-rvlnd   0/1     Pending   0          0s    <none>   kn5    <none>           <none>
marco@kn1:~$ kubectl get pods -l run=my-nginx -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE   NOMINATED NODE   READINESS GATES
my-nginx-df7bbf6f5-bxh54   1/1     Running   0          46s   172.17.0.2   kn4    <none>           <none>
my-nginx-df7bbf6f5-rvlnd   1/1     Running   0          46s   172.17.0.2   kn5    <none>           <none>

[Tej-Singh-Rana]

Hi @marco74 ,

If you choose the service type: ClusterIP then you can curl the nginx service from any pods (curl pkg should be available on that pod's image or later on, you can install it by its package manager) on that XYZ node in your cluster.
Suppose pod-1 is deployed on worker-01 node and pod-2 is deployed on worker-02 node
then you can Curl to that nginx service on its <CLUSTER-IP>:<PORT> from inside pod (pod-1 or pod-2).
Simply run the command: kubectl exec -it pod-1 -- bash
then do curl <CLUSTER-IP>:<PORT>

you can do the same from pod-2 as well. It will work the same.

If you want to curl from the node then change service type: ClusterIP to NodePort.

Reference link - https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport

Regards,

[marco74]

@Tej-Singh-Rana: Thank you for that hint! I can test this tomorrow. Just to give quicker replies (and hopefully answers): Isn't that considered as alpha in 1.24? So I'm wondering, why https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/ is not just working as expected. This also does not answer the question why two pods get the same IP. As far as I know networking, networking cannot work properly, when two participants have the same IP.

Regards,

[marco74]

After lot of testing I a stack running on Debian Buster. This works really stable. It is also possible to join other nodes to the stack and run pod on that stack. Pods on different nodes also have different IPs as expected.

Since the current version (since August 2021) is Debian Bullseye (Debian 11), i also run the same instructions there. As a comparison:

Debian Buster:

marco@kn1:~$ kubectl get pods -n kube-system
NAME                          READY   STATUS    RESTARTS   AGE
coredns-6d4b75cb6d-8rx4g      0/1     Pending   0          45h
coredns-6d4b75cb6d-f5mqt      0/1     Pending   0          45h
etcd-kn1                      1/1     Running   0          45h
kube-apiserver-kn1            1/1     Running   0          45h
kube-controller-manager-kn1   1/1     Running   0          45h
kube-proxy-77m89              1/1     Running   0          45h
kube-scheduler-kn1            1/1     Running   0          45h

Debian Bullseye:

marco@kn4:~$ kubectl get pods -n kube-system
NAME                          READY   STATUS             RESTARTS          AGE
coredns-6d4b75cb6d-2qz2m      0/1     Pending            0                 45h
coredns-6d4b75cb6d-l5b5g      0/1     Pending            0                 45h
etcd-kn4                      1/1     Running            325 (8m25s ago)   45h
kube-apiserver-kn4            1/1     Running            420 (21m ago)     45h
kube-controller-manager-kn4   1/1     Running            468 (7m36s ago)   45h
kube-scheduler-kn4            0/1     CrashLoopBackOff   487 (97s ago)     45h

You can easily reproduce by taking Virtualbox, setting up two VMs, one with Buster on with Bullseye and use the same instructions I used for dozens of times:

sudo apt-get -y update
sudo apt-get -y install ca-certificates curl gnupg lsb-release
sudo mkdir -p /opt/bin/
sudo curl --location https://github.com/flannel-io/flannel/releases/download/v0.18.0/flanneld-amd64 -o /opt/bin/flanneld
sudo chmod +x /opt/bin/flanneld #/usr/local/bin/flanneld
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get -y update
sudo apt-get -y install containerd.io
sudo sed -i "s/^\(disabled_plugins\s*=\)/# \1/" /etc/containerd/config.toml
cat <<EOF | sudo tee -a /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true
EOF

sudo systemctl restart containerd
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo kubeadm config images pull
sudo kubeadm --pod-network-cidr=10.244.0.0/16 init
kubectl get pods -n kube-system

Regards,

[marco74]

After tests with Ubuntu 20.04 and 22.04 I found out following. Kubernetes stops supporting newer distributions including Ubuntu 22.04 and Debian 11 with Version 1.21. Since that version kube-system pods get restarted constantly:

marco@kn3:~$ cat /etc/debian_version 
11.3
marco@kn3:~$ dpkg -l kubectl kubeadm kubelet
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=====================================
ii  kubeadm        1.21.13-00   amd64        Kubernetes Cluster Bootstrapping Tool
ii  kubectl        1.21.13-00   amd64        Kubernetes Command Line Tool
ii  kubelet        1.21.13-00   amd64        Kubernetes Node Agent
marco@kn3:~$ kubectl get pods --all-namespaces
NAMESPACE     NAME                          READY   STATUS    RESTARTS   AGE
kube-system   coredns-558bd4d5db-dqb6t      0/1     Pending   0          57m
kube-system   coredns-558bd4d5db-p65tb      0/1     Pending   0          56m
kube-system   etcd-kn3                      1/1     Running   15         57m
kube-system   kube-apiserver-kn3            1/1     Running   13         57m
kube-system   kube-controller-manager-kn3   1/1     Running   13         57m
kube-system   kube-proxy-76dvp              1/1     Running   14         57m
kube-system   kube-scheduler-kn3            1/1     Running   12         57m

I assume that it's not intended to restart pods that often. Just as a comparision tha same with version 1.20 on the same OS (different VM):

11.3
marco@kn2:~$ dpkg -l kubectl kubeadm kubelet
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=====================================
ii  kubeadm        1.20.15-00   amd64        Kubernetes Cluster Bootstrapping Tool
ii  kubectl        1.20.15-00   amd64        Kubernetes Command Line Tool
ii  kubelet        1.20.15-00   amd64        Kubernetes Node Agent
marco@kn2:~$ kubectl get pods --all-namespaces
NAMESPACE     NAME                          READY   STATUS    RESTARTS   AGE
kube-system   coredns-74ff55c5b-tbjxw       0/1     Pending   0          11h
kube-system   coredns-74ff55c5b-xb4ws       0/1     Pending   0          11h
kube-system   etcd-kn2                      1/1     Running   3          11h
kube-system   kube-apiserver-kn2            1/1     Running   2          11h
kube-system   kube-controller-manager-kn2   1/1     Running   2          11h
kube-system   kube-proxy-l8rwl              1/1     Running   2          11h
kube-system   kube-scheduler-kn2            1/1     Running   2          11h

The only difference in setting up the stack was

sudo apt-get -y install kubelet=1.21.13-00 kubectl=1.21.13-00 kubeadm=1.21.13-00

vs.

sudo apt-get -y install kubelet=1.20.15-00 kubeadm=1.20.15-00 kubectl=1.20.15-00

So, what prevents newer kubernetes versions from working cleanly on the latest distributions Debian 11 and Ubuntu 22.04? Does anybody know how to get kubernetes working on those distributions as well?

Regards,

[marco74]

After having a look at kubernetes/kubernetes#105762 I found a solution, that solves the restart problem on Debian 11. I do not use Ubuntu, therefore feel free to also test it on Ubuntu 22.04.

With the hint I updated /etc/containerd/config.toml:

version = 2

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

With this I got the stack running without restarts, also the network initialization works very smooth. My suggestion for the kubernetes documentation team. could you please add the two lines

• version = 2
• runtime_type = "io.containerd.runc.v2"

at https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd ? I was not able to get the stack running without those two lines.

For all those who are interested in how I finally setup a cluster with Debian 11 and kubernetes 1.24:

sudo apt-get -y update
sudo apt-get -y install ca-certificates curl gnupg lsb-release

#from:
# * https://gist.github.com/dbathgate/737e8d800decde9adbf8a0130870bcc4
# * https://github.com/flannel-io/flannel
sudo mkdir -p /opt/bin/
sudo curl --location https://github.com/flannel-io/flannel/releases/download/v0.18.0/flanneld-amd64 -o /opt/bin/flanneld
sudo chmod +x /opt/bin/flanneld #/usr/local/bin/flanneld

curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo   "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get -y update

sudo apt-get -y install containerd.io
sudo sed -i "s/^\(disabled_plugins\s*=\)/# \1/" /etc/containerd/config.toml
cat <<EOF | sudo tee -a /etc/containerd/config.toml
### from https://github.com/kubernetes/kubernetes/issues/105762
version = 2

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

EOF
sudo systemctl restart containerd

## Prerequisites for kubernetes, according to:
##   https://kubernetes.io/docs/setup/production-environment/container-runtimes/
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

## install kubernetes
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl

sudo kubeadm config images pull

sudo kubeadm --pod-network-cidr=10.244.0.0/16 init

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

kubectl get pods -n kube-system -o wide

Regards,

[sftim]

@marco74 as of now, is there a task page that needs fixing? If so, what's the task page and what in that page isn't right?

[marco74]

@sftim Thank you for asking, just looked up:

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd describes how to setup containerd for Kubernetes.

When reading containerd's documentation for configuration (https://github.com/containerd/cri/blob/master/docs/config.md) they write about version = 2. This was added with commit e2550f6285638aaf591d40927c319a301b697d5f on Jul, 24 2019 (containerd/cri@e2550f6), not fully 3 years ago. runtime_type = "io.containerd.runc.v2" is mentioned in the same documentation since containerd/cri@aaddaa2, on Dec, 16 2019. This was about 2.5 years ago.

Debian 11 released about 10 months ago. It works quite stable but I could not get Kubernetes running without those two settings. I also tried omitting one of the two lines – both are needed. I tested without using those two lines on Ubuntu 22.04 and could not get it running either. I did not test on Ubuntu 22.04, but I am pretty sure that this will work with those two lines.

It would be great, if somebody could explain me why version 2 is necessary, currently I do not know it. At the moment I only know that it helps. As you can read in this thread, I read about this on another thread, After I looked why the system-containers are restarting continuously.

Therefore I would really appreciate if this block:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

could be adapted to this:

version = 2

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

Could you, or probably somebody else do that?

Marco

[sftim]

This looks mainly as if it's an query about how to set up a container runtime for a Kubernetes cluster with working DNS and networking. That's a tricky aspect to document because Kubernetes is only one part of that story; the other three parts are: a CNI-compatible network plugin, a CRI-compatible container runtime, and a cluster DNS implementation such as CoreDNS.

Based on the situation as I see it:
/retitle Help troubleshooting a cluster deployed using kubeadm
/kind support

We'd really like to see this kind of detail covered in the documentation for containerd and not the documentation for Kubernetes. I notice that this report mentions two different CRI implementations (containerd and cri-dockerd). Kubernetes as a project doesn't provide support for any container runtime, and that of course means we don't promise to offer help for troubleshooting Docker Engine, we don't promise to offer help for troubleshooting cri-dockerd, and we don't promise to offer help for troubleshooting containerd.

If you have a similar problem in future, try asking on Server Fault. That's a dedicated online community for advice about this kind of problem.

If you've got the appetite for it @marco74, you could file a feature request against https://github.com/containerd/containerd.io - or even make an introduction there and see if there are any ways you can help document the thing. Optional of course!