Skip to content

[Bug]: Flaky timing assertion in WaitForVMRunning integration test #56

Description

@mrhillsman

Bug Description

The integration test WaitForVMRunning / when waiting with a very short timeout / should return error quickly in internal/testutil/testutil_integration_test.go is flaky. It asserts that a 1-second timeout completes in under 3 seconds, but on cold runs (first test execution after build) the Kubernetes client's API discovery, TLS handshake, and connection setup add enough overhead to push elapsed time past the 3-second ceiling.

Steps to Reproduce

  1. Build the project fresh (clear any test cache): go clean -testcache
  2. Run integration tests: go test -count=1 -tags integration ./internal/testutil/...
  3. Observe the test may fail on the first run but pass on subsequent runs.

Expected Behavior

The test should pass reliably on both cold and warm runs.

Actual Behavior

On cold runs, WaitForVMRunning with a 1s timeout takes ~3.3s due to Kubernetes client cold-start overhead (API discovery, TLS negotiation), exceeding the < 3*time.Second assertion:

• [FAILED] [6.110 seconds]
WaitForVMRunning when waiting with a very short timeout [It] should return error quickly

  [FAILED] Expected
      <time.Duration>: 3323609895
  to be <
      <time.Duration>: 3000000000

On subsequent (warm) runs, the connection pool and discovery cache are reused and the test passes.

Root cause

Two factors:

  1. Tight upper-bound assertion — 3s leaves no room for first-call client overhead against a real cluster.
  2. No client warm-up — the test's BeforeEach creates the client but doesn't make a throwaway API call, so the first real call inside WaitForVMRunning pays the full cold-start cost.

Proposed fix

  • Add a throwaway API call in BeforeEach to warm the client connection before timing-sensitive tests.
  • Widen the upper bound to < 5*time.Second to accommodate residual variance.

Command

  • Other

Workload Type

  • Not applicable

Go Version

go1.25.0 linux/amd64

Deployment Mode

  • CLI from local machine (kubeconfig)

Additional Context

Observed on Fedora 43 (kernel 7.0.9-104.fc43.x86_64). The flake is timing-dependent and more likely on slower machines or clusters with higher API latency.

Metadata

Metadata

Assignees

No one assigned

    Labels

    good first issueDenotes an issue ready for a new contributor, according to the "help wanted" guidelines.kind/bugCategorizes issue or PR as related to a bug.kind/flakeCategorizes issue or PR as related to a flaky test.size/XSDenotes a PR that changes 0-9 lines, ignoring generated files.triage/acceptedIndicates an issue or PR is ready to be actively worked on.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions