[epic] Secure communication with x509 certificates

[kvaps]

Secure communication between etcd members and etcd clients with x509 certificates.

This is a last task remained before we start using etcd-operator in our cozystack platform.

• Implement mounting user certificates and keys #125

[kvaps]

George Gaál (@gecube) could you please elaborate on this, and write user stories and user spec for certificates

[kvaps]

link https://github.com/aenix-io/etcd-operator/pull/87/files

[Kirill-Garbar]

How to create certificates infrastructure

Permalink to kamaji-etcd repo.

cfssl gencert -initca /csr/ca-csr.json | cfssljson -bare /certs/ca &&
mv /certs/ca.pem /certs/ca.crt && mv /certs/ca-key.pem /certs/ca.key &&
cfssl gencert -ca=/certs/ca.crt -ca-key=/certs/ca.key -config=/csr/config.json -profile=peer-authentication /csr/peer-csr.json | cfssljson -bare /certs/peer &&
cfssl gencert -ca=/certs/ca.crt -ca-key=/certs/ca.key -config=/csr/config.json -profile=peer-authentication /csr/server-csr.json | cfssljson -bare /certs/server &&
cfssl gencert -ca=/certs/ca.crt -ca-key=/certs/ca.key -config=/csr/config.json -profile=client-authentication /csr/root-client-csr.json | cfssljson -bare /certs/root-client

Generate CA

cfssl gencert -initca /csr/ca-csr.json | cfssljson -bare /certs/ca - initiates CA key and certificate. We need to create ca-csr configuration. Kamaji example:

{
  "CN": "Clastix CA",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "IT",
      "ST": "Italy",
      "L": "Milan"
    }
  ]
}

CSR for our etcd (if we need it with cert-manager):

{
  "CN": "AEnix etcd cluster <<cluster-name>> CA",
  "key": {
    "algo": "rsa",
    "size": 4096
  },
  "names": [
    {
      "C": "?",
      "ST": "?",
      "L": "?"
    }
  ]
}

Important points:

• Reissue CA certificate automatically.
• Size 4096 bytes.
• We can't rotate CA key. If CA key is rotated, then all client and peer certificates must be signed by new CA key at the same time => possible long downtime. Old certificates signed by old CA key will not work.
• Different CAs for peer and client authentication. Some customers will need to issue their own certificates for clients applications and will not want to touch peer certificates.
• We should not place ca.key to etcd-cluster namespace and mount it to etcd cluster (kamaji mounts it to etcd-cluster, defrag and backup jobs). Ca.key is necessary only for issuing peer and client certificates - can be placed to etcd-operator namespace and must be protected.
• Separate CA for every etcd cluster.
  • Why: Want to make etcd-clusters isolated. No need for different etcd clusters to have common CA.
  • Possible risk: Will applications be able to use different certificates to different etcd clusters if they need? Do libraries allow that?

Questions:

• How to fill C, ST, L fields? Are they mandatory?

Generate certificate for peer authentication

Peer authentication certs authenticate etcd nodes against etcd nodes and used as server certificates.

cfssl gencert -ca=/certs/ca.crt -ca-key=/certs/ca.key -config=/csr/config.json -profile=peer-authentication /csr/peer-csr.json | cfssljson -bare /certs/peer

config.json - configuration for certificate requests (1 year validity):

{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "server-authentication": {
        "usages": ["signing", "key encipherment", "server auth"],
        "expiry": "8760h"
      },
      "client-authentication": {
        "usages": ["signing", "key encipherment", "client auth"],
        "expiry": "8760h"
      },
      "peer-authentication": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}

peer-csr.json - certificate request config. We need to create it. Kamaji example:

{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 4096
  },
  "hosts": ["test-test-etcd-0",
    "test-test-etcd-0.test-test-etcd",
    "test-test-etcd-0.test-test-etcd.etcd-operator-system.svc",
    "test-test-etcd-0.test-test-etcd.etcd-operator-system.svc.cluster.local","test-test-etcd-1",
    "test-test-etcd-1.test-test-etcd",
    "test-test-etcd-1.test-test-etcd.etcd-operator-system.svc",
    "test-test-etcd-1.test-test-etcd.etcd-operator-system.svc.cluster.local","test-test-etcd-2",
    "test-test-etcd-2.test-test-etcd",
    "test-test-etcd-2.test-test-etcd.etcd-operator-system.svc",
    "test-test-etcd-2.test-test-etcd.etcd-operator-system.svc.cluster.local",
    "127.0.0.1"
  ]
}

• test-test-etcd - helm release name
• etcd-operator-system - namespace name
• 3 replicas

Hostnames helm template:

{{- range $count := until (int $.Values.replicas) -}}
    {{ printf "\"%s-%d\"," ( include "etcd.stsName" $outer ) $count }}
    {{ printf "\"%s-%d.%s\"," ( include "etcd.stsName" $outer ) $count (include "etcd.serviceName" $outer) }}
    {{ printf "\"%s-%d.%s.%s.svc\"," ( include "etcd.stsName" $outer ) $count (include "etcd.serviceName" $outer) $.Release.Namespace }}
    {{ printf "\"%s-%d.%s.%s.svc.cluster.local\"," ( include "etcd.stsName" $outer ) $count (include "etcd.serviceName" $outer) $.Release.Namespace }}
{{- end }}

Important points:

• Rotate private keys. They are not rotated by cert-manager by default. If we enable this feature, private key and certificate will be written at the same time => no downtime is expected. See cert-manager docs.

Questions:

• Key length? It is not recommended to use 2048, but 4096 can bring higher CPU usage. 3072? Cert-manager does not recreate private key if key length setting is changed. It was true before - most probably now length can be changed on-the-fly.
• Expiration period? Hardcoded 1 year for the first implementation (30 days reissue before expiration).

Generate server certificate for client trust

cfssl gencert -ca=/certs/ca.crt -ca-key=/certs/ca.key -config=/csr/config.json -profile=peer-authentication /csr/server-csr.json | cfssljson -bare /certs/server

server-csr.json - certificate request config. We need to create it. Kamaji example:

{
 "CN": "etcd",
 "key": {
   "algo": "rsa",
   "size": 4096
 },
 "hosts": ["test-test-etcd-0.test-test-etcd.etcd-operator-system.svc.cluster.local","test-test-etcd-1.test-test-etcd.etcd-operator-system.svc.cluster.local","test-test-etcd-2.test-test-etcd.etcd-operator-system.svc.cluster.local",
   "etcd-server.etcd-operator-system.svc.cluster.local",
   "etcd-server.etcd-operator-system.svc",
   "etcd-server",
   "127.0.0.1"
 ]
}

Hostnames helm template:

{{- range $count := until (int $.Values.replicas) -}}
    {{ printf "\"%s-%d.%s.%s.svc.cluster.local\"," ( include "etcd.fullname" $outer ) $count (include "etcd.serviceName" $outer) $.Release.Namespace }}
{{- end }}

Important points:

• Rotate private keys. They are not rotated by cert-manager by default. If we enable this feature, private key and certificate will be written at the same time => no downtime is expected. See cert-manager docs.

Questions:

• Key length? It is not recommended to use 2048, but 4096 can bring higher CPU usage. 3072? Cert-manager does not recreate private key if key length setting is changed. It was true before - most probably now length can be changed on-the-fly.
• Expiration period? Hardcoded 1 year for the first implementation (30 days reissue before expiration).

Generate root client certificate

cfssl gencert -ca=/certs/ca.crt -ca-key=/certs/ca.key -config=/csr/config.json -profile=client-authentication /csr/root-client-csr.json | cfssljson -bare /certs/root-client

root-client-csr.json request config. We need to create it. Kamaji example:

{
  "CN": "root",
  "key": {
    "algo": "rsa",
    "size": 4096
  },
  "names": [
    {
      "O": "system:masters"
    }
  ]
}

Important points:

• Root certificate will be used only by operator for maintenance tasks. Certificate for Kubernetes is generated separately.
• Rotate private keys. They are not rotated by cert-manager by default. If we enable this feature, private key and certificate will be written at the same time => no downtime is expected. See cert-manager docs.
• Root client certificate can be used for kubernetes api server application during the beta test period.

Questions:

• Key length? It is not recommended to use 2048, but 4096 can bring higher CPU usage. 3072? Cert-manager does not recreate private key if key length setting is changed. It was true before - most probably now length can be changed on-the-fly.
• Expiration period? Hardcoded 1 year for the first implementation (30 days reissue before expiration).

Etcd reference for v3.5

What certificates and private keys to use for what configuration key: kamaji-etcd chart:

Client-to-server communication:

--cert-file=<path>: Certificate used for SSL/TLS connections to etcd. When this option is set, advertise-client-urls can use the HTTPS schema.

--key-file=<path>: Key for the certificate. Must be unencrypted.

--client-cert-auth: When this is set etcd will check all incoming HTTPS requests for a client certificate signed by the trusted CA, requests that don’t supply a valid client certificate will fail. If authentication is enabled, the certificate provides credentials for the user name given by the Common Name field.

--trusted-ca-file=<path>: Trusted certificate authority.

--auto-tls: Use automatically generated self-signed certificates for TLS connections with clients.

Peer (server-to-server / cluster) communication:

The peer options work the same way as the client-to-server options:

--peer-cert-file=<path>: Certificate used for SSL/TLS connections between peers. This will be used both for listening on the peer address as well as sending requests to other peers.

--peer-key-file=<path>: Key for the certificate. Must be unencrypted.

--peer-client-cert-auth: When set, etcd will check all incoming peer requests from the cluster for valid client certificates signed by the supplied CA.

--peer-trusted-ca-file=<path>: Trusted certificate authority.

--peer-auto-tls: Use automatically generated self-signed certificates for TLS connections between peers.

If either a client-to-server or peer certificate is supplied the key must also be set. All of these configuration options are also available through the environment variables, ETCD_CA_FILE, ETCD_PEER_CA_FILE and so on.

Common options:

--cipher-suites: Comma-separated list of supported TLS cipher suites between server/client and peers (empty will be auto-populated by Go).

--tls-min-version=<version> Sets the minimum TLS version supported by etcd.

--tls-max-version=<version> Sets the maximum TLS version supported by etcd. If not set the maximum version supported by Go will be used.

[Kirill-Garbar]

For now it was decided to create secret reference sections in the etcdCluster spec and expect secrets with certificates from the user of etcd-operator.

Basic spec from Timofei Larkin (@lllamnyp) (TBD)

spec:
  security:
    serverTLSSecretRef: # secretRef
      name: server-tls-secret
    clientCertAuth: true # bool
    trustedCAFile: # secretRef
      name: trusted-tls-secret
    peerTLSSecretRef: # secretRef
      name: peer-tls-secret
    peerClientCertAuth: true # bool

FYI Andrei Kvapil (@kvaps)

[Kirill-Garbar]

Peer.ca

Option	Description
secretName	Secret name of user-provided secret. If not specified then operator generates certificate by the spec below
metadata	Metadata of generated secret.
duration	Expiration time of generated secret.
renewBefore	Time period before expiration time when certificate will be reissued.
privateKey	Private key configuration: algorithm and key size.

Peer.cert

Option	Description
secretName	Secret name of user-provided secret. If not specified then operator generates certificate by the spec below. If peer.ca.secretName is provided, then this certificate is generated from the CA that was provided by the user. You can't define the secret name in this section and do not define peer.ca.secretName.
metadata	Metadata of generated secret.
duration	Expiration time of generated secret.
renewBefore	Time period before expiration time when certificate will be reissued.
privateKey	Private key configuration: algorithm, key size and boolean parameter is it necessary to rotate private key when certificate is expired

ClientServer section has the same fields as peer section.

Rbac

Option	Description
enabled	Enables role-based access control: creates root user in etcd, gives him root role and enables authentication in etcd.

spec:
  security:
    peer:
      enabled: true # optional
      ca:
        # if not defined, then operator generates CA by the spec below
        secretName: ext-peer-ca-tls-secret # oneof secretName or secretTemplate
        secretTemplate: # oneof secretName or secretTemplate
          annotations: {} # optional
          labels: {} # optional
        duration: 86400h # optional
        renewBefore: 720h # optional
        privateKey:
          algorithm: RSA # optional
          size: 4096 # optional
      cert:
        secretName: ext-peer-tls-secret
        secretTemplate:
          annotations: {}
          labels: {}
        duration: 720h
        renewBefore: 180h
        privateKey:
          rotate: true # optional
          algorithm: RSA
          size: 4096
    clientServer:
      enabled: true
      ca:
        secretName: ext-server-ca-tls-secret
        secretTemplate:
          annotations: {}
          labels: {}
        duration: 86400h
        renewBefore: 720h
        privateKey:
          algorithm: RSA
          size: 4096
      serverCert:
        secretName: ext-server-tls-secret
        secretTemplate:
          annotations: {}
          labels: {}
        extraSANs: []
        duration: 720h
        renewBefore: 180h
        privateKey:
          rotate: true
          algorithm: RSA
          size: 4096
      rootClientCert:
        secretName: ext-client-tls-secret
        secretTemplate:
          annotations: {}
          labels: {}
        duration: 720h
        renewBefore: 180h
        privateKey:
          rotate: true
          algorithm: RSA
          size: 4096
      auth:
        enabled: true # optional

Important points:

• If field has a value and it is optional, then this value is a default.
• peer:
  • If ca.secretName is not defined, operator generates its own CA.
  • If ca.secretName is defined, then every field under secretName should not be defined.
  • If cert.secretName id not defined, then certificate is generate by operator from the CA defined in the section above (user-managed or operator-managed).
  • User must define ca.secretName if cert.secretName is defined.
  • Algorithm is a list of the values. NOTE: look into the lib that generates certs what values exist (or to cert-manager).
• clientServer:
  • See peer logic.
  • RootClientCert uses server ca and has the same logic as server.cert.
  • Rbac.enabled enables role-based access control: creates root user in etcd, gives him root role and enables authentication in etcd.

[kvaps]

I would suggest to reuse logic of resource templates we introduced before, eg for every specific resources configuration:

secretTemplate:
  metadata:
    name: ""
    annotations: {}
    labels: {}

Instead of this:

secretName: ""
metadata:
  annotations: {}
  labels: {}