Handle post-failover pod replacement gracefully

[ysqyang]

Summary

When Valkey promotes a replica to primary during automatic failover and Kubernetes recreates the old node-index=0 pod, the reconciler previously tried to assign a new slot range to the replacement pod. This failed in a loop with "no slots range to assign" because all 16384 slots were already owned by the promoted replica, leaving the cluster stuck in Degraded/NodeAddFailed.

Fix: before assigning slots, assignSlotsToPendingPrimaries now checks the live Valkey topology (shardExistsInTopology) for any existing member in the same shard. If one exists (post-failover or mid-failover), the replacement pod joins as a replica instead. replicateToShardPrimary also gains a fallback (findShardPrimary) that scans all shard pods for the actual primary, since after failover node-index=0 is no longer the Valkey primary.

What changed

• utils.go — two new helpers:
  • shardHasLivePrimary: checks if another pod in the same shard-index group is a slot-bearing primary in the live topology
  • findShardPrimary: scans all pods in a shard to find the actual Valkey primary regardless of node-index label
• valkeycluster_controller.go:
  • addValkeyNode step 2 now detects post-failover replacements and falls through to the replica path
  • replicateToShardPrimary tries node-index=0 first (fast path), then falls back to findShardPrimary

Test plan

1. Deploy a 3-shard, 1-replica cluster and wait for Ready

kubectl apply -f config/samples/v1alpha1_valkeycluster.yaml

2. Kill shard-0 primary

kubectl delete pod valkeycluster-sample-0-0

3. Verify valkeycluster-sample-0-1 is now the primary and valkeycluster-sample-0-0 is its replica

kubectl exec valkeycluster-sample-0-1 -- valkey-cli cluster nodes

[jdheyburn]

On the whole the code looks sound, but I will let the better Cluster experts comment first

[bjosv]

Is the argument selfAddress really needed? It's an optimization to save a few cpu cycles right?
The api gets cleaner if we remove it.
Same comment for shardExistsInTopology

[ysqyang]

Good catch. I just realized that both functions are called with a pending node's address, which can't possibly appear in state.Shards (it wasn't part of any shard when the state was captured).

[bjosv]

The comment needs to be updated here, shardHasLivePrimary does not exist in this PR.

[bjosv]

..also update the PR description with the latest changes, shardHasLivePrimary is mentioned there too.

[ysqyang]

done

[bjosv]

LGTM, I have run the new testcase multiple times (20+) without failure.
We'll see if we can catch it again.

[bjosv]

We should remove the pictures when merging to avoid messy git history. Also remove what changed since its mentions shardHasLivePrimary.
I'll merge if you are ok with this PR @sandeepkunusoth ?