Add an assertions concept for infra specs, with Node.js example

.editorconfig

@@ -0,0 +1,7 @@
+root = true
+
+[*]
+charset = utf-8
+trim_trailing_whitespace = true
+indent_style = space
+indent_size = 2

Dockerfile

@@ -0,0 +1,2 @@
+# This is just a dummy while the actual builds are run by ./hooks/build
+FROM ubuntu:20.04@sha256:8bce67040cd0ae39e0beb55bcb976a824d9966d2ac8d2e4bf6119b45505cee64

README.md

@@ -1,11 +1,78 @@
-# Basic Kubernetes monitoring
+# Kubernetes Assert
 
-... as Kustomize base.
-The scope here is to set up basic monitoring using only `kubectl` as tooling.
+### WIP status: Search for `?)` in this text to find pending decisions. Search for TODO to find pending work.
 
-## Apply
+Kubernetes Assert is our re-think of [build-contract](https://github.com/Yolean/build-contract), for Kubernetes.
+As usual we don't write tooling. We combine mainstream stuff that we find lean enough.
 
-Assuming that Prometheus Operator is already installed,
+Summary:
+
+1. If you don't have a monitoring stack already, use the one from this repo.
+   It's the Prometheus setup we use in dev clusters.
+2. Arrange your specs like in [our example](./runtime-nodejs/example-project/) with a [skaffold.yaml](./runtime-nodejs/example-project/skaffold.yaml) and [kustomization.yaml](./runtime-nodejs/example-project/kustomization.yaml).
+   - Our near term roadmap is to support "just put your spec files here" to run but we're not [there](./runtime-nodejs/example-specs/) yet. You need a [Dockerfile](./runtime-nodejs/example-project/Dockerfile).
+   - You need a way to [build](https://skaffold.dev/docs/pipeline-stages/builders/) with Skaffold. Our examples use [y-stack](https://github.com/y-stack/ystack) which is completely local, but any build method is fine.
+3. Run `skaffold dev`
+4. Make sure Prometheus will [scrape `assertions_failed`](./assertions_failed/).
+5. Watch for Alerts using for example
+   `kubectl -n monitoring exec alertmanager-main-0 -c alertmanager -- wget -qO- http://127.0.0.1:9093/api/v2/alerts | jq`
+   or the web interface, or a pager.
+
+## What is a test
+
+The idea is that any container that exports an `assertions_failed` counter (OR gauge?) is a test.
+Obviously the metric needs to be scraped, and your team must be alerted about any(?) non-zeroness.
+Integration test results are inherently less binary than unit tests. You may want to tweak the parameters for alerting, maybe on a per test basis.
+
+## The dev loop
+
+There won't be any tests if the dev loop is unpleasant.
+Our idea of a good dev loop is that while you're editing your specs, they re-run on save and you'll see the test output.
+
+The goal however is that tests run unattended, and reliably call for your attention when they fail.
+At that point you'll want to know which test suite and test run that failed, and see the output.
+
+Assumptions:
+
+* Tests must run in-cluster. That's what they'll do during CI.
+  - Running them locally while developing would cause "works for me" issuess.
+* Skaffold is a good enough dev loop tool.
+* Because Skaffold is the dev loop tool, `skaffold run` is the CI tool.
+* The test environment is one of:
+  - A local ephemeral cluster that starts empty when your test starts.
+  - A local dev cluster like [y-stack](https://github.com/y-stack/ystack).
+  - A temporary namespace in a test cluster with production-like backends.
+  - An ephemeral kubernetes-in-kubernetes cluster managed by your CI pipeline. Could be [kind](https://github.com/kubernetes-sigs/kind).
+* ... so as the test author you should simply assume that whatever is in your [skaffold.yaml](https://skaffold.dev/docs/references/yaml/)'s `deploy` section has been applied (you'll do the waiting), and the given context's current namespace is all yours.
+  - TODO There'll be RBAC for read access, including access to logs.
+
+We recommend using a _runtime_ for tests to a) minimize bolerplate and b) aid "common code ownership" for specs by enforcing a structure.
+
+### The Node.js Jest runtime
+
+Our first runtime for Kubernets Assert is based on [Jest](https://jestjs.io/):
+
+* BDD style
+* Specs don't need to be compiled, we can copy source to the runtime.
+* The test runner will translate Jest results to `assertions_failed`. Specs are free to [export](https://www.npmjs.com/package/prom-client#counter) other metrics of any kind.
+* By design we'll always run in [watch](https://jestjs.io/docs/en/cli#--watchall) mode. If specs are static that's equal to one test run but staying alive to keep exporting metrics.
+   - Success criterias and cleanup is implemented by the pipeline.
+
+How are specs delivered to the runtime?
+
+1. If your specs are a "project", i.e. have a package.json, you need a proper build step. Use the runtime as base image. Install to `/usr/src/specs`(?). See example TODO.
+   - Note that Jest `--watch` (i.e. the runtime's `skaffold dev`) requires source to be in a git repo. A `git init` with no commits is fine, but remember to include your `.gitignore`.
+1. If your specs are fine with the [runtime's](./runtime-nodejs/package.json) `dependencies`
+   (feel free to have utility .js files alongside specs) they need to be mounted or copied to `/usr/src/specs/src`(?)
+
+How to avoid boilerplate?
+
+* You still need yaml. But the actual workflow definition can be inherited from the runtime's [Kustomize base](./runtime-nodejs/base/) `- github.com/Yolean/kubernetes-assert/runtime-nodejs/kustomize/?ref=[your choice]`.
+* Create your [kustomization.yaml](https://kubectl.docs.kubernetes.io/pages/reference/kustomize.html) then run `skaffold init`. See example TODO.
+
+## Apply the example monitoring stack
+
+Assuming that `github.com/coreos/prometheus-operator/?ref=[a recent revision]` is already installed,
 start from the example kustomize base:
 
 ```
@@ -17,6 +84,14 @@ kubectl apply -k grafana
 Note how [Prometheus](./example-small/now-prometheus.yaml) will match rules and monitors
 using the label(s) that the [kustomization.yaml](./example-small/kustomization.yaml) adds.
 
+## Your Kustomize base
+
+A real stack might start from example-small and then:
+
+- Change the replicas count for prometheus and alertmanager
+- Change the retention for prometheus `now` to a bit longer
+- Add aggergation and long-term storage, presumably using [Thanos](https://thanos.io/)
+
 ## Re-generate stuff
 
 This repo needs to have some generated content, where upstream kustomize bases could not be found
@@ -25,10 +100,37 @@ This repo needs to have some generated content, where upstream kustomize bases c
 docker-compose -f docker-compose.test.yml up --build kubernetes-mixin
 ```
 
-## Storage classes
+## CI test suite
+
+WIP
+
+```
+docker volume rm kubernetes-monitoring_admin 2> /dev/null || true
+./test.sh
+```
+
+## Development
+
+WIP.
+We tend to use [y-stack](https://github.com/y-stack/ystack) but when working with test (in particular failing ones) it might help to reuse the CI stack.
+
+```
+compose='docker-compose -f docker-compose.test.yml -f docker-compose.dev-overrides.yml'
+$compose down \
+  ;docker volume rm kubernetes-monitoring_admin kubernetes-monitoring_k3s-server 2>/dev/null || true
+sudo rm test/.kube/kubeconfig.yaml
+$compose up -d sut
+export KUBECONFIG=$PWD/test/.kube/kubeconfig.yaml
+```
 
-Per cluster, for example
+## Releases
 
 ```
-kubectl -n monitoring apply -f storageclasses-gke/
+git push
+# wait for https://hub.docker.com/r/solsson/kubernetes-assert, then
+YOLEAN_PROMOTE=true IMAGE_NAME=solsson/kubernetes-assert:latest ./hooks/build
+# update examples so that people get started from refs
+grep -A 1 bases: runtime-nodejs/example-project/kustomization.yaml
+grep FROM runtime-nodejs/example-project/Dockerfile
+# validate examples
 ```

assertions_failed/kubernetes-assert-podmonitor.yaml

@@ -0,0 +1,14 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: kubernetes-assert-metrics
+  # Note that this excludes labels; see ../base-label-prometheus-now/
+spec:
+  jobLabel: kubernetes-assert
+  namespaceSelector:
+    any: true
+  selector:
+    matchLabels:
+      kubernetes-assert: metrics
+  podMetricsEndpoints:
+  - port: prometheus

assertions_failed/kustomization.yaml

@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+- kubernetes-assert-podmonitor.yaml

base-label-prometheus-now/kustomization.yaml

@@ -7,3 +7,4 @@ bases:
 - ../node-exporter
 - ../kube-state-metrics
 - ../kubernetes-mixin
+- ../assertions_failed

docker-compose.dev-overrides.yml

@@ -0,0 +1,27 @@
+version: '2.4'
+services:
+
+  server:
+    ports:
+    - 17243:17243
+
+  agent:
+    ports:
+    # Failed to get traefik to work in docker-compose; using a predefined nodeport range instead
+    #- 80:80
+    #- 443:443
+    - "31720-31729:31720-31729"
+    mem_limit: 8589934592
+    memswap_limit: 0
+
+  sut:
+    environment:
+    - KEEP_RUNNING=true
+
+volumes:
+  admin:
+    driver: local
+    driver_opts:
+      type: none
+      device: $PWD/test
+      o: bind

docker-compose.test.yml

@@ -1,6 +1,86 @@
 version: '2.4'
 services:
 
+  server:
+    image: rancher/k3s:v1.17.5-k3s1@sha256:3025ecfa3c7c3a262a97435cb7c9c182d22b93d014bac767019d65bdce586786
+    command:
+    - server
+    - --https-listen-port=17243
+    - --disable-agent
+    - --node-name=server
+    - --no-deploy=traefik
+    - --no-deploy=servicelb
+    - --kube-apiserver-arg
+    -   service-node-port-range=31720-31729
+    tmpfs:
+    - /run
+    - /var/run
+    privileged: true
+    environment:
+    - K3S_TOKEN=somethingtotallyrandom
+    - K3S_KUBECONFIG_OUTPUT=/admin/.kube/kubeconfig.yaml
+    - K3S_KUBECONFIG_MODE=666
+    expose:
+    - 17243
+    - 8472
+    - 10250
+    volumes:
+    - k3s-server:/var/lib/rancher/k3s
+    - admin:/admin
+    mem_limit:  300000000
+    memswap_limit: 0
+
+  agent:
+    depends_on:
+    - server
+    image: rancher/k3s:v1.17.5-k3s1@sha256:3025ecfa3c7c3a262a97435cb7c9c182d22b93d014bac767019d65bdce586786
+    tmpfs:
+    - /run
+    - /var/run
+    privileged: true
+    environment:
+    - K3S_URL=https://server:17243
+    - K3S_TOKEN=somethingtotallyrandom
+    expose:
+    - 8472
+    - 10250
+    mem_limit: 1500000000
+    memswap_limit: 0
+
+  sut:
+    depends_on:
+    - server
+    - agent
+    image: solsson/ystack-runner@sha256:95aff82cb5245e93c8f3193253b054a5628f6421ddc5f93b5dfff0cf38d1bbc7
+    environment:
+    - PROMETHEUS_OPERATOR_BASE=github.com/coreos/prometheus-operator/?ref=9877ec897fc7ea4f50d947ca824ae0c30c185105
+    - KUBECONFIG_WAIT=30
+    - KEEP_RUNNING=false
+    volumes:
+    - admin:/admin
+    entrypoint:
+    - /bin/bash
+    - -cx
+    command:
+    - |
+      mkdir ~/.kube
+      until test -f /admin/.kube/kubeconfig.yaml; do
+        [ $$KUBECONFIG_WAIT -gt 0 ] || exit ${BULID_EXIT_CODE_ON_NO_CLUSTER:-0}
+        KUBECONFIG_WAIT=$$(( $$KUBECONFIG_WAIT - 1 ))
+        echo "Waiting for a kubeconfig ..." && sleep 1
+      done
+      set -e
+      cat /admin/.kube/kubeconfig.yaml | sed 's|127.0.0.1|server|' > ~/.kube/config
+      kubectl-waitretry --for=condition=Ready node --all
+      kubectl get namespace monitoring 2>/dev/null || kubectl create namespace monitoring
+      kubectl apply -k $$PROMETHEUS_OPERATOR_BASE
+      if [ "$$KEEP_RUNNING" = "true" ]; then
+        echo "Will stay running for manual work"
+        sleep infinity
+      fi
+    mem_limit:   80000000
+    memswap_limit: 0
+
   kubernetes-mixin:
     build: ./kubernetes-mixin
     environment:
@@ -45,13 +125,6 @@ services:
     - ./kubernetes-mixin:/kustomize-base
     - ./kubernetes-mixin-dashboards:/dashboards-kustomize-base
 
-  sut:
-    image: golang:1.14.3-buster@sha256:6edcc1af51905a860d510a7d4bb0a83d9a0e381cf05c07f23722020a4c688e6f
-    #environment:
-    #- DEBUG=true
-    entrypoint:
-    - bash
-    - -ce
-    command:
-    - |
-      echo "No tests"
+volumes:
+  k3s-server: {}
+  admin: {}

grafana/grafana-deployment.yaml

@@ -14,7 +14,7 @@ spec:
     spec:
       containers:
       - name: grafana
-        image: grafana/grafana:7.0.0-ubuntu@sha256:d4fe3f23dcf9e106419eed2573607fa09e5160475ac4e9ba9021f8f79073efba
+        image: grafana/grafana:7.0.3-ubuntu@sha256:1bfb488dce8b54c4e438f626648efc98c14ab64ca9a833e441b7200334376acc
         env:
         # Has to be a kustomization
         # - name: GF_SERVER_ROOT_URL

hooks/build

@@ -0,0 +1,48 @@
+#!/bin/bash
+set -e
+
+[ -z "$IMAGE_NAME" ] && echo "IMAGE_NAME is required" && exit 1;
+
+function imagename {
+  buildname=$1
+  case $IMAGE_NAME in
+    *:latest) echo -n $IMAGE_NAME | sed "s|:latest|:$buildname|" ;;
+    *:*) echo -n $IMAGE_NAME | sed "s|:\(.*\)|:\1-$buildname|" ;;
+    *) echo $IMAGE_NAME:$buildname ;;
+  esac
+}
+
+if [ "$YOLEAN_PROMOTE" != "true" ]; then
+  docker build -t $IMAGE_NAME .
+else
+  YOLEAN_PROMOTE_TAG="$(curl -L 'https://hub.docker.com/api/audit/v1/build/?include_related=true&offset=0&limit=10&object=%2Fapi%2Frepo%2Fv1%2Frepository%2Fsolsson%2Fkubernetes-assert%2F' | jq -r '.objects[0].commit')"
+  echo "Latest build's git ref is $YOLEAN_PROMOTE_TAG, right?"
+  read -p "Verified? [y/n] " ok && [ "$ok" = "y" ] || exit 1
+fi
+
+CONTEXTS="
+runtime-nodejs
+specs
+"
+
+PUSH=""
+
+for CONTEXT in $CONTEXTS; do
+
+  CONTEXT_IMAGE_NAME=$(imagename $CONTEXT)
+
+  if [ "$YOLEAN_PROMOTE" != "true" ]; then
+    docker build -t $CONTEXT_IMAGE_NAME ./$CONTEXT
+    PUSH="$PUSH $CONTEXT_IMAGE_NAME"
+  else
+    YOLEAN_PROMOTE_NAME=yolean/assert-$CONTEXT:$YOLEAN_PROMOTE_TAG
+    docker pull $CONTEXT_IMAGE_NAME
+    docker tag $CONTEXT_IMAGE_NAME $YOLEAN_PROMOTE_NAME
+    docker tag $CONTEXT_IMAGE_NAME yolean/assert-$CONTEXT:latest
+    PUSH="$PUSH $YOLEAN_PROMOTE_NAME yolean/assert-$CONTEXT:latest"
+  fi
+done
+
+echo "PUSH list contains: $PUSH"
+[ -z "$NOPUSH" ] || exit 0
+for P in $PUSH; do docker push $P; done

kube-state-metrics/kube-state-metrics-deployment.yaml

@@ -20,7 +20,7 @@ spec:
         - --port=8081
         - --telemetry-host=0.0.0.0
         - --telemetry-port=8082
-        image: quay.io/coreos/kube-state-metrics:v1.9.6@sha256:7baf75cfa9f70966f2d1f9e724edfb8970fd0ce817fc59887f650cf83b92a14d
+        image: quay.io/coreos/kube-state-metrics:v1.9.7@sha256:2f82f0da199c60a7699c43c63a295c44e673242de0b7ee1b17c2d5a23bec34cb
         name: kube-state-metrics
         ports:
         - name: web