test_proxy_deployment.bats failed

[JeffreyDallas]

Because bug #866 ,
BATS tests have NOT been running for a long time,

if running manually, one of tests is failing now

Running BATS: '/Users/jeffrey/full-stack-testing/charts/fullstack-deployment/tests/../../../dev/bats/bats-core/bin/bats /Users/jeffrey/full-stack-testing/charts/fullstack-deployment/tests'
=============================================================
test_basic_deployment.bats
 ✓ Check all network node pods are running
 ✓ Check systemctl is running in all root containers
test_gateway_api_deployment.bats
 ✓ Check Network Node GRPC routes
test_proxy_deployment.bats
 ✗ Check haproxy deployment
   tags: deployment-test
   (in test file test_proxy_deployment.bats, line 45)
     `[ "${test_status}" = "${PASS}" ]' failed
   --------------------------Env Setup: fullstack-testing Helm Test------------------------------------------------
   NAMESPACE: fst-jeffrey
   RELEASE_NAME: fst
   ENV_FILE: /Users/jeffrey/full-stack-testing/charts/fullstack-deployment/tests/.env
   BATS_HOME: /Users/jeffrey/full-stack-testing/charts/fullstack-deployment/tests/../../../dev/bats
   TESTS_DIR: /Users/jeffrey/full-stack-testing/charts/fullstack-deployment/tests
   LOG: /tmp/fullstack-testing-logs/helm-test.log
   OUTPUT_LOG: false
   -----------------------------------------------------------------------------------------------------
   
   --------------------------Env Setup: fullstack-testing Helm Test------------------------------------------------
   NAMESPACE: fst-jeffrey
   RELEASE_NAME: fst
   ENV_FILE: /Users/jeffrey/full-stack-testing/charts/fullstack-deployment/tests/.env
   BATS_HOME: /Users/jeffrey/full-stack-testing/charts/fullstack-deployment/tests/../../../dev/bats
   TESTS_DIR: /Users/jeffrey/full-stack-testing/charts/fullstack-deployment/tests
   LOG: /tmp/fullstack-testing-logs/helm-test.log
   OUTPUT_LOG: false
   -----------------------------------------------------------------------------------------------------
   
   Loading bats helper...
   Loading test helper scripts...
 ✓ Check envoy proxy deployment
test_sidecar_deployment.bats
 ✓ Check record-stream-uploader sidecar
 ✓ Check event-stream-uploader sidecar
 ✓ Check account-balance-uploader sidecar
 ✓ Check backup-uploader sidecar
 ✓ Check otel-collector sidecar

10 tests, 1 failure

Steps :

  cd dev/
 make setup setup-cluster deploy-network

  cd ../charts/fullstack-deployment/tests/
  ./run.sh 

[JeffreyDallas]

k9s shows haproxy-node0 and haproxy-node2 are not ready

 Events:                                                                                                                                                                                                                           │
│   Type     Reason     Age                    From               Message                                                                                                                                                           │
│   ----     ------     ----                   ----               -------                                                                                                                                                           │
│   Normal   Scheduled  9m3s                   default-scheduler  Successfully assigned fst-jeffrey/haproxy-node0-86844c889c-5bkh9 to fst-control-plane                                                                             │
│   Normal   Pulling    9m2s                   kubelet            Pulling image "docker.io/haproxytech/haproxy-alpine:2.4.25"                                                                                                       │
│   Normal   Pulled     8m37s                  kubelet            Successfully pulled image "docker.io/haproxytech/haproxy-alpine:2.4.25" in 497.0945ms (24.676460595s including waiting)                                           │
│   Normal   Created    8m35s (x2 over 8m37s)  kubelet            Created container haproxy                                                                                                                                         │
│   Normal   Started    8m35s (x2 over 8m37s)  kubelet            Started container haproxy                                                                                                                                         │
│   Normal   Pulled     8m35s                  kubelet            Container image "docker.io/haproxytech/haproxy-alpine:2.4.25" already present on machine                                                                          │
│   Warning  Unhealthy  7m32s                  kubelet            Startup probe failed: jq not found, installing jq                                                                                                                 │
│ fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/aarch64/APKINDEX.tar.gz                                                                                                                                                    │
│ fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/aarch64/APKINDEX.tar.gz                                                                                                                                               │
│ (1/2) Installing oniguruma (6.9.8-r1)                                                                                                                                                                                             │
│ (2/2) Installing jq (1.6-r4)                                                                                                                                                                                                      │
│ Executing busybox-1.36.1-r5.trigger                                                                                                                                                                                               │
│ OK: 15 MiB in 24 packages                                                                                                                                                                                                         │
│ http_backend status: DOWN                                                                                                                                                                                                         │
│   Warning  Unhealthy  3m57s (x43 over 7m27s)  kubelet  Startup probe failed: http_backend status: DOWN 

[jeromy-cannon]

@JeffreyDallas , I would have to look at this more, but just to let you know, per Nathan's request HaProxy will now show as running phase as soon as it comes up. However, it won't show PodCondition[Ready]=true until after it sees that the network node has opened up the grpc port, which is close to the time that it shows ACTIVE in its logs.

[JeffreyDallas]

I tried multiple times, sometimes all three haproxy could not be ready due to http_backend status: DOWN
sometimes two out of three haproxy has http_backend status: DOWN

[jeromy-cannon]

Yes, the HaProxy will show as Ready 0/1, Phase(Status) = Running, until the grpc port is active on the network node.

So, it will look like this:

until after the network nodes are in ACTIVE in the swirlds.log

[jeromy-cannon]

It is checking the condition:

function is_pod_ready() {
  local pod=$1
  [[ -z "${pod}" ]] && echo "ERROR: Pod name is needed (is_pod_ready)" && return "${EX_ERR}"

  local pod_status=$(kubectl get pod "${pod}" -o jsonpath="{.status.conditions[?(@.type=='Ready')].status}" -n "${NAMESPACE}" | tr '[:lower:]' '[:upper:]')
  [ -z "${pod_status}" ] && pod_status="FALSE"

  log_debug "Pod '${pod}' is ready in namespace ${NAMESPACE} : ${pod_status}"

  [[ "${pod_status}" = "TRUE" ]] && return "${EX_OK}"
  return "${EX_ERR}"
}

this will fail until after the network node software has been installed and the network comes up.

the haproxy check needs to change to do an is_pod_running, and not check the podCondition, but instead check the phase. You can see the difference in how solo is checking for running vs ready.

example:

  async checkNetworkNodePod (namespace, nodeId, maxAttempts = 60, delay = 2000) {
    nodeId = nodeId.trim()
    const podName = Templates.renderNetworkPodName(nodeId)

    try {
      await this.k8.waitForPods([constants.POD_PHASE_RUNNING], [
        'fullstack.hedera.com/type=network-node',
        `fullstack.hedera.com/node-name=${nodeId}`
      ], 1, maxAttempts, delay)

      return podName
    } catch (e) {
      throw new FullstackTestingError(`no pod found for nodeId: ${nodeId}`, e)
    }
  }