Skip to content

Flaky DNS from worker nodes during bootc-operator e2e tests #59

@jlebon

Description

@jlebon

Hit an instance in GHA where cri-o on the worker node wasn't able to resolve the registry.

From the logs (yay that we now have logs!):

 === Pod status ===
 NAME                                                 READY   STATUS             RESTARTS   AGE     IP             NODE                              NOMINATED NODE   READINESS GATES
 bootc-operator-controller-manager-57dcb4847d-jpd7t   1/1     Running            0          4m14s   192.168.49.3   controller                        <none>           <none>
 bootc-operator-daemon-pk5fs                          0/1     ImagePullBackOff   0          3m2s    10.85.0.2      testcontrollermembership-27fd3b   <none>           <none>

 === Events ===
 LAST SEEN   TYPE      REASON              OBJECT                                                    MESSAGE
 4m15s       Normal    SuccessfulCreate    replicaset/bootc-operator-controller-manager-57dcb4847d   Created pod: bootc-operator-controller-manager-57dcb4847d-jpd7t
 4m15s       Normal    Pulling             pod/bootc-operator-controller-manager-57dcb4847d-jpd7t    Pulling image "registry.cluster.local:5000/bootc-operator-e2e:latest"
 4m15s       Normal    ScalingReplicaSet   deployment/bootc-operator-controller-manager              Scaled up replica set bootc-operator-controller-manager-57dcb4847d from 0 to 1
 4m15s       Normal    Scheduled           pod/bootc-operator-controller-manager-57dcb4847d-jpd7t    Successfully assigned bootc-operator/bootc-operator-controller-manager-57dcb4847d-jpd7t to controller
 4m10s       Normal    Created             pod/bootc-operator-controller-manager-57dcb4847d-jpd7t    Container created
 4m10s       Normal    Started             pod/bootc-operator-controller-manager-57dcb4847d-jpd7t    Container started
 4m10s       Normal    Pulled              pod/bootc-operator-controller-manager-57dcb4847d-jpd7t    Successfully pulled image "registry.cluster.local:5000/bootc-operator-e2e:latest" in 4.51s (4.511s including waiting). Image size: 284085022 bytes.
 4m10s       Normal    LeaderElection      lease/bootc-operator.bootc.dev                            bootc-operator-controller-manager-57dcb4847d-jpd7t_52210329-9a57-4eeb-8c36-0725505fdff3 became leader
 3m3s        Normal    Scheduled           pod/bootc-operator-daemon-pk5fs                           Successfully assigned bootc-operator/bootc-operator-daemon-pk5fs to testcontrollermembership-27fd3b
 3m3s        Normal    SuccessfulCreate    daemonset/bootc-operator-daemon                           Created pod: bootc-operator-daemon-pk5fs
 81s         Normal    Pulling             pod/bootc-operator-daemon-pk5fs                           Pulling image "registry.cluster.local:5000/bootc-operator-e2e:latest"
 81s         Warning   Failed              pod/bootc-operator-daemon-pk5fs                           Failed to pull image "registry.cluster.local:5000/bootc-operator-e2e:latest": unable to pull image: initializing source docker://registry.cluster.local:5000/bootc-operator-e2e:latest: pinging container registry
 registry.cluster.local:5000: Get "http://registry.cluster.local:5000/v2/": dial tcp: lookup registry.cluster.local: no such host
 81s         Warning   Failed              pod/bootc-operator-daemon-pk5fs                           Error: ErrImagePull
 5s          Normal    BackOff             pod/bootc-operator-daemon-pk5fs                           Back-off pulling image "registry.cluster.local:5000/bootc-operator-e2e:latest"
 5s          Warning   Failed              pod/bootc-operator-daemon-pk5fs                           Error: ImagePullBackOff

GHA run: https://github.com/jlebon/bootc-operator/actions/runs/27290067829/job/80608104645

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions