certificate expired and rotate

[liyongxian]

Hi
By default,k3s components' certificates is 1 year,those components include:kube-apiserver,scheduler,cloud-controller,K3s should rotate those certificates.

K3s version: v1.17.4+k3s1

A clear and concise description of what you want to happen.

The components' certificates expire,k3s will not work.
Thanks.

[Bowser1704]

Hi
It bothers me, too.
Do you have a good way to update the certificate automatically?|
Thanks.

[brandond]

Certificates rotate automatically if they are <90 days from expiration when k3s is started. This has been the case since v0.10: #805

As long as you are patching and restarting your k3s infrastructure at least every couple months, you shouldn't run into expired certificates.

[Bowser1704]

@brandond
Does it mean we have to restart k3s if we want to update certificate?
Thanks.

[brandond]

Yes. Make sure you're running at least 0.10, and restart k3s. If the certs are less than 90 days from expiration they will be rotated when it starts. I don't believe there's a way to manually rotate them or make them valid for longer... other than poking at the files manually with a 3rd party tool like openssl of course.

You're restarting your k3s nodes at least every couple months to apply OS patches, right?

[Bowser1704]

@brandond
OK Thanks.

[ieugen]

Since the question has been answered, maybe the issue can be closed. cc @liyongxian .

[davidnuzik]

Closing the issue. If there are further questions let us know and we can reopen if necessary.

[ethinx]

Hi @brandond ,

I did a test on VM, modify system time to the day before expired time ( < 90 days) and restart k3s, then checking cert info by using openssl command as following, found the cert has no change.

openssl s_client -connect localhost:6443 -showcerts < /dev/null 2>&1 | openssl x509 -noout -enddate

I tried reset the time several times even clean up the certs in /var/lib/rancher/k3s/server/tls, but unlucky, k3s unable to start up. It seems that somewhere cached the certificates?

Jun  2 17:15:06 localhost k3s: E0602 17:15:06.593592    7212 reflector.go:153] k8s.io/client-go@v1.17.5-k3s1/tools/cache/reflector.go:105: Failed to list *v1.VolumeAttachment: Get "https://127.0.0.1:6444/apis/storage.k8s.io/v1/volumeattachments?limit=500&resourceVersion=0": x509: certificate has expired or is not yet valid: current time 2020-06-02T17:15:06Z is before 2022-04-26T23:02:59Z

Could you please advise if there is any misunderstanding?

Version: v1.17.5+k3s1

Thanks,

[brandond]

@ethinx did you set the time into the future, start k3s, and then set it back again? According to the error, your cert was generated almost two years in the future and is not yet valid:

x509: certificate has expired or is not yet valid: current time 2020-06-02T17:15:06Z is before 2022-04-26T23:02:59Z

[ethinx]

@brandond since I tried to clean up the certs under /var/lib/rancher/server/tls and screw things up.

Make it simpler, I did a fresh install and test again as following, do I miss something here?

# INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh

# openssl s_client -connect localhost:6443 -showcerts </dev/null 2>&1 | openssl x509 -noout -startdate -enddate
notBefore=Jun  3 13:27:43 2020 GMT
notAfter=Jun  3 13:27:43 2021 GMT

[root@localhost ~]# date -s 20210520
Thu May 20 00:00:00 UTC 2021
[root@localhost ~]# hwclock -w

[root@localhost ~]# date
Thu May 20 00:00:08 UTC 2021

# systemctl restart k3s

# openssl s_client -connect localhost:6443 -showcerts </dev/null 2>&1 | openssl x509 -noout -startdate -enddate
notBefore=Jun  3 13:27:43 2020 GMT
notAfter=Jun  3 13:27:43 2021 GMT

# k3s -v
k3s version v1.17.5+k3s1 (58ebdb2a)

[brandond]

@ethinx Don't set the time forward while it's running. If you're going to test it that way, stop k3s before changing the system time. Also, make sure you've disabled any time sync daemons - ntpd, chrony, systemd-timesyncd, etc.

[ethinx]

Hi @brandond,

It seems that not about the time sync daemons, even though I stop chronyd (I didn't notice it before), cert keep unchanged after I stop k3s -> set time to future -> start k3s.

I think I find where the cert be cached, it's the k3s-serving secret under namespace kube-system.

The cert get updated after I remove the secret and local cache file dynamic-cert.json. You could check the procedures in my gist: https://gist.github.com/ethinx/c0ede41a7a6a340681476ef564ea75f2

Just have a glance on the code, seems that the local file certificate generation is different to the secret.

Hope you could share more about it, as we don't have documents about this.

[brandond]

I think you might be correct. @galal-hussein @erikwilson it looks like the kube-apiserver, kubelet, and auth-proxy certs all call through certutil.LoadOrGenerateKeyFile which handles rotation, but the dynamiclistener/factory stuff that creates the k3s-serving key has its own CA loader and cert generation code that does not appear to check expiration or handle renewal.