Skip to content

Support external redis in helm chart #12010

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mik-laj merged 2 commits into from
Jan 9, 2021
Merged

Support external redis in helm chart #12010

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1ddd164
Fix chart network policies definion and compliance
Oct 31, 2020
87ada94
Support external redis instance in helm chart
Oct 31, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 34 additions & 7 deletions chart/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,8 +207,10 @@ The following tables lists the configurable parameters of the Airflow chart and
| `extraConfigMaps` | Extra ConfigMaps that will be managed by the chart | `{}` |
| `data.metadataSecretName` | Secret name to mount Airflow connection string from | `~` |
| `data.resultBackendSecretName` | Secret name to mount Celery result backend connection string from | `~` |
| `data.brokerUrlSecretName` | Secret name to mount redis connection url string from | `~` |
| `data.metadataConection` | Field separated connection data (alternative to secret name) | `{}` |
| `data.resultBackendConnection` | Field separated connection data (alternative to secret name) | `{}` |
| `data.brokerUrl` | String containing the redis broker url (if you are using an "external" redis) | `{}` |
| `fernetKey` | String representing an Airflow Fernet key | `~` |
| `fernetKeySecretName` | Secret name for Airflow Fernet key | `~` |
| `kerberos.enabled` | Enable kerberos support for workers | `false` |
Expand All @@ -218,7 +220,7 @@ The following tables lists the configurable parameters of the Airflow chart and
| `kerberos.keytabPath` | Path for the Kerberos keytab file | `/etc/airflow.keytab` |
| `kerberos.principal` | Name of the Kerberos principal | `airflow` |
| `kerberos.reinitFrequency` | Frequency of reinitialization of the Kerberos token | `3600` |
| `kerberos.config` | Content of the configuration file for kerberos (might be templated using Helm templates) | `<see values.yaml>` |
| `kerberos.config` | Content of the configuration file for kerberos (might be templated using Helm templates) | `<see values.yaml>` |
| `workers.replicas` | Replica count for Celery workers (if applicable) | `1` |
| `workers.keda.enabled` | Enable KEDA autoscaling features | `false` |
| `workers.keda.pollingInverval` | How often KEDA should poll the backend database for metrics in seconds | `5` |
Expand All @@ -234,9 +236,9 @@ The following tables lists the configurable parameters of the Airflow chart and
| `workers.resources.requests.memory` | Memory Request of workers | `~` |
| `workers.terminationGracePeriodSeconds` | How long Kubernetes should wait for Celery workers to gracefully drain before force killing | `600` |
| `workers.safeToEvict` | Allow Kubernetes to evict worker pods if needed (node downscaling) | `true` |
| `workers.serviceAccountAnnotations` | Annotations to add to worker kubernetes service account | `{}` |
| `workers.extraVolumes` | Mount additional volumes into worker | `[]` |
| `workers.extraVolumeMounts` | Mount additional volumes into worker | `[]` |
| `workers.serviceAccountAnnotations` | Annotations to add to worker kubernetes service account | `{}` |
| `workers.extraVolumes` | Mount additional volumes into worker | `[]` |
| `workers.extraVolumeMounts` | Mount additional volumes into worker | `[]` |
| `workers.nodeSelector` | Node labels for pod assignment | `{}` |
| `workers.affinity` | Affinity labels for pod assignment | `{}` |
| `workers.tolerations` | Toleration labels for pod assignment | `[]` |
Expand All @@ -249,9 +251,9 @@ The following tables lists the configurable parameters of the Airflow chart and
| `scheduler.resources.requests.memory` | Memory Request of scheduler | `~` |
| `scheduler.airflowLocalSettings` | Custom Airflow local settings python file | `~` |
| `scheduler.safeToEvict` | Allow Kubernetes to evict scheduler pods if needed (node downscaling) | `true` |
| `scheduler.serviceAccountAnnotations` | Annotations to add to scheduler kubernetes service account | `{}` |
| `scheduler.extraVolumes` | Mount additional volumes into scheduler | `[]` |
| `scheduler.extraVolumeMounts` | Mount additional volumes into scheduler | `[]` |
| `scheduler.serviceAccountAnnotations` | Annotations to add to scheduler kubernetes service account | `{}` |
| `scheduler.extraVolumes` | Mount additional volumes into scheduler | `[]` |
| `scheduler.extraVolumeMounts` | Mount additional volumes into scheduler | `[]` |
| `scheduler.nodeSelector` | Node labels for pod assignment | `{}` |
| `scheduler.affinity` | Affinity labels for pod assignment | `{}` |
| `scheduler.tolerations` | Toleration labels for pod assignment | `[]` |
Expand Down Expand Up @@ -282,6 +284,18 @@ The following tables lists the configurable parameters of the Airflow chart and
| `pgbouncer.nodeSelector` | Node labels for pod assignment | `{}` |
| `pgbouncer.affinity` | Affinity labels for pod assignment | `{}` |
| `pgbouncer.tolerations` | Toleration labels for pod assignment | `[]` |
| `redis.enabled` | Enable the redis provisioned by the chart | `true` |
| `redis.terminationGracePeriodSeconds` | Grace period for tasks to finish after SIGTERM is sent from Kubernetes. | `600` |
| `redis.persistence.enabled` | Enable persistent volumes. | `true` |
| `redis.persistence.size` | Volume size for redis StatefulSet. | `1Gi` |
| `redis.persistence.storageClassName` | If using a custom storageClass, pass name ref to all StatefulSets here. | `default` |
| `redis.resources.limits.cpu` | CPU Limit of redis | `~` |
| `redis.resources.limits.memory` | Memory Limit of redis | `~` |
| `redis.resources.requests.cpu` | CPU Request of redis | `~` |
| `redis.resources.requests.memory` | Memory Request of redis | `~` |
| `redis.passwordSecretName` | Redis password secret. | `~` |
| `redis.password` | If password is set, create secret with it, else generate a new one on install. | `~` |
| `redis.safeToEvict` | This setting tells Kubernetes that its ok to evict when it wants to scale a node down. | `true` |
| `redis.nodeSelector` | Node labels for pod assignment | `{}` |
| `redis.affinity` | Affinity labels for pod assignment | `{}` |
| `redis.tolerations` | Toleration labels for pod assignment | `[]` |
Expand Down Expand Up @@ -334,6 +348,19 @@ helm install airflow . \
--set workers.persistence.enabled=false
```

## Using an external redis instance

When using the `CeleryExecutor` or the `CeleryKubernetesExecutor` the chart will by default create a redis Deployment/StatefulSet alongside airflow.
You can also use "your own" redis instance by providing the `data.brokerUrl` (or `data.borkerUrlSecretName`) value directly:

```bash
helm install airflow . \
--namespace airflow \
--set executor=CeleryExecutor \
--set redis.enabled=false \
--set data.brokerUrl=redis://redis-user:password@redis-host:6379/0
```

## Walkthrough using kind

**Install kind, and create a cluster:**
Expand Down
2 changes: 1 addition & 1 deletion chart/templates/_helpers.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@
- name: AIRFLOW__CELERY__BROKER_URL
valueFrom:
secretKeyRef:
name: {{ .Release.Name }}-broker-url
name: {{ default (printf "%s-broker-url" .Release.Name) .Values.data.brokerUrlSecretName }}
key: connection
{{- end }}
{{- if .Values.elasticsearch.enabled }}
Expand Down
51 changes: 51 additions & 0 deletions chart/templates/check-values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

{{- /*
The sole purpose of this yaml file is it to check the values file is consistent for some complexe combinations.
*/ -}}

{{- /*
##############################
Redis related checks
#############################
*/ -}}

{{- if or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor") }}
{{- if .Values.redis.enabled }}

{{- if and .Values.redis.passwordSecretName (not .Values.data.brokerUrlSecretName) }}
{{ required "When using the internal redis of the chart and setting the value redis.passwordSecretName, you must also set the value data.brokerUrlSecretName." nil }}
{{- end }}

{{- if and .Values.redis.passwordSecretName .Values.redis.password }}
{{ required "You must not set both values redis.passwordSecretName and redis.password" nil }}
{{- end }}

{{- else }}

{{- if not (or .Values.data.brokerUrlSecretName .Values.data.brokerUrl) }}
{{ required "You must set one of the values data.brokerUrlSecretName or data.brokerUrl when using a Celery based executor with redis.enabled set to false (we need the url to the redis instance)." nil }}
{{- end }}

{{- end }}

{{- if and .Values.data.brokerUrlSecretName .Values.data.brokerUrl }}
{{ required "You must not set both values data.brokerUrlSecretName and data.brokerUrl" nil }}
{{- end }}

{{- end }}
6 changes: 3 additions & 3 deletions chart/templates/flower/flower-ingress.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,10 @@ metadata:
release: {{ .Release.Name }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: {{ .Release.Service }}
{{- with .Values.ingress.flower.annotations }}
annotations:
{{- range $key, $value := .Values.ingress.flower.annotations }}
{{ $key }}: {{ $value | quote }}
{{- end }}
{{ toYaml . | indent 4 }}
{{- end }}
spec:
{{- if .Values.ingress.flower.tls.enabled }}
tls:
Expand Down
14 changes: 7 additions & 7 deletions chart/templates/flower/flower-networkpolicy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,13 +40,13 @@ spec:
component: flower
release: {{ .Release.Name }}
policyTypes:
- Ingress
ingress:
- from:
- Ingress
{{- if .Values.flower.extraNetworkPolicies }}
{{ toYaml .Values.flower.extraNetworkPolicies | indent 4 }}
ingress:
- from:
{{ toYaml .Values.flower.extraNetworkPolicies | indent 6 }}
ports:
- protocol: TCP
port: {{ .Values.ports.flowerUI }}
{{- end }}
ports:
- protocol: TCP
port: {{ .Values.ports.flowerUI }}
{{- end }}
2 changes: 1 addition & 1 deletion chart/templates/redis/redis-networkpolicy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
################################
## Airflow Redis NetworkPolicy
#################################
{{- if (and .Values.networkPolicies.enabled (or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor"))) }}
{{- if and .Values.redis.enabled .Values.networkPolicies.enabled (or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor")) }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
Expand Down
2 changes: 1 addition & 1 deletion chart/templates/redis/redis-service.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
################################
## Airflow Redis Service
#################################
{{- if or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor") }}
{{- if and .Values.redis.enabled (or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor")) }}
kind: Service
apiVersion: v1
metadata:
Expand Down
2 changes: 1 addition & 1 deletion chart/templates/redis/redis-statefulset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
################################
## Airflow Redis StatefulSet
#################################
{{- if or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor") }}
{{- if and .Values.redis.enabled (or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor")) }}
{{- $nodeSelector := or .Values.redis.nodeSelector .Values.nodeSelector }}
{{- $affinity := or .Values.redis.affinity .Values.affinity }}
{{- $tolerations := or .Values.redis.tolerations .Values.tolerations }}
Expand Down
Loading