Feat/tier aware upgrade ordering by AdheipSingh · Pull Request #17 · apache/druid-operator

AdheipSingh · 2026-05-04T20:06:41Z

Fixes #XXXX.

Description

This PR has:

been tested on a real K8S cluster to ensure creation of a brand new Druid cluster works.
been tested for backward compatibility on a real K*S cluster by applying the changes introduced here on an existing Druid cluster. If there are any backward incompatible changes then they have been noted in the PR description.
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added documentation for new or modified features or behaviors.

Key changed/added files in this PR

MyFoo
OurBar
TheirBaz

AdheipSingh · 2026-05-04T20:16:09Z

Note: The following is review is purely written by me and not edited or reviewed or taken any inspiration by any AI.

Problem Statement

The issue raised is valid, as the operator reconciles multiple druid nodes based on purely nodeType. The underlying structure is map[string]nodeSpec, within nodeSpec the only way to segregate nodes is based on nodeType. So if i have 10 historical with X,Y,Z names, they all will get upgraded at same time, which is a pure disruption.

I would propose for an implementation which fixes the problem at its core, and sticks to druid native design on how it represents nodeTypes and tiers.

Approach

If i break down the main abstractions, we are dealing with 4 constructs. ( nodeType is present as of now, i plan to introduce order of upgrade, tiers and order of upgrade of tiers ).

OrderOfUpgrade
OrderOfUpgradeOfTiers
NodeTypes
Tiers

As a user i should be able to define the following for my druid nodes.

i want to upgrade my druid nodes in a certain order defined.
i want to define tiers within my druid nodes.
i want to define the order of upgrades of my tiers for each druid node defined.

Tiers

In druid we can categorize historicals as well as brokers into tiers. A tier which defines separate groups of Historicals and Brokers to receive different query assignments / loading rules etc.

How does this map in the operator spec ?

We introduce tier as a key within the nodeSpec, scoped same with nodeType.

historical-aws-az1:
    tier: hot
    nodeType: historical
historical-aws-az2:
    tier: cold
    nodeType:historical
broker-aws-az1:
    tier: hot
    nodeType: broker
broker-aws-az1:
    tier: cold
    nodeType: broker
.....

OrderOfUpgrade

Remove the hardcoded order in the code, though the order is based on druid's recommendation, a lot of times i had a custom order of upgrade. The main for loop should construct an order based on this.

orderOfUpgrade:
 - historical
 - broker
 - middlemanager
 - coordinator
 - router
 - overlord

This is a []string structure and on each reconcile should be built.

OrderOfUpgradeOfTiers

WIthin a nodeType we need to have the ability to define order of upgrade of nodeTypes. So that needs to be defined as a separate structure for each tier

OrderOfUpgradeOfTiers:

- histroricals:
     - hot
     - cold
     - glacier
- broker
     - hot
     - cold

Here's a combined spec.

spec:
  orderOfUpgrade:
    - historical
    - broker
    - middleManager
    - coordinator
    - router
    - overlord
  
  orderOfUpgradeOfTiers:
    historical:
      - hot
      - cold
      - glacier
    broker:
      - hot
      - cold
  
  nodes:
    historical-aws-az1:
      tier: hot
      nodeType: historical
    
    historical-aws-az2:
      tier: cold
      nodeType: historical
    
    historical-aws-az3:
      tier: glacier
      nodeType: historical
    
    broker-aws-az1:
      tier: hot
      nodeType: broker
    
    broker-aws-az2:
      tier: cold
      nodeType: broker
    
    middlemanagers:
      nodeType: middleManager
    
    coordinators:
      nodeType: coordinator
    
    routers:
      nodeType: router
    
    overlords:
      nodeType: overlord

Upgrade execution:

historical:hot → historical-aws-az1
historical:cold → historical-aws-az2
historical:glacier → historical-aws-az3
broker:hot → broker-aws-az1
broker:cold → broker-aws-az2
middleManager → middlemanagers
coordinator → coordinators
router → routers
overlord → overlords

Implementation

Signature remains as it is

func getNodeSpecsByOrder(m *v1alpha1.Druid) []*ServiceGroup {
    // Returns ordered list of ServiceGroups ready for sequential rollout
}

The existing service group should be extended to use :

type ServiceGroup struct {
    key      string                  
    nodeType string                      
    tier     string                      
    spec     v1alpha1.DruidNodeSpec      
}

An ideal service group after construction should look like this:

[
    {key: "historical-aws-az1", nodeType: "historical", tier: "hot", spec: ...},
    {key: "historical-aws-az2", nodeType: "historical", tier: "cold", spec: ...},
    {key: "historical-aws-az3", nodeType: "historical", tier: "glacier", spec: ...},
    {key: "broker-az1", nodeType: "broker", tier: "hot", spec: ...},
    {key: "broker-az2", nodeType: "broker", tier: "cold", spec: ...},
    {key: "mm-1", nodeType: "middleManager", tier: "", spec: ...},
]

This way we solve the problem at the core. I have half of the implementation already done, and would like to raise a PR and get feedback. Also this is needs to be backward compatible, so we make sure regardless of user specifying the above we fallback to the default/current way.

AdheipSingh · 2026-05-04T20:27:07Z

+	return result
+}
+
+func sortServiceGroups(groups []*ServiceGroup, tierOrder []string) {


sortServiceGroups sorts nodes of the same node type to determine their rollout order. tierRank map will convert the tier order slice ["hot", "cold", "glacier"] into a lookup: {"hot": 0, "cold": 1, "glacier": 2}.

aruraghuwanshi · 2026-05-09T20:07:08Z

 func getNodeSpecsByOrder(m *v1alpha1.Druid) []*ServiceGroup {
+	nodeTypeOrder := defaultDruidServicesOrder
+	if len(m.Spec.OrderOfUpgrade) > 0 {
+		nodeTypeOrder = m.Spec.OrderOfUpgrade


Could we make this path a little more defensive?

If orderOfUpgrade is set, it looks like the reconciler only returns node groups whose nodeType appears in that list. For a partial list, typo, or unknown node type, configured nodes could be skipped entirely.

Since deployDruidCluster uses this returned list to populate the resource name maps used by cleanup, skipped node groups may later look unused and be deleted.

Maybe we could either validate orderOfUpgrade as a complete/valid list for the configured node types, or treat it as a priority list and append any remaining configured node types in the default deterministic order.

aruraghuwanshi · 2026-05-09T20:11:55Z

+				return err == nil
+			}, timeout, interval).Should(BeTrue())
+		})
+		It("Should return nodes ordered by custom nodeType order and tier order", func() {


Could we add a small regression test around incomplete or invalid orderOfUpgrade inputs?

The happy path for tier ordering is covered, but the more risky behavior seems to be partial, unknown, or duplicate node type entries. A test showing that remaining configured nodes are still reconciled, or that the spec is rejected during validation, would make this safer to evolve.

aruraghuwanshi · 2026-05-10T17:09:23Z

+	// OrderOfUpgrade defines the order in which node types are upgraded during a rolling deploy.
+	// If not specified, the default order is used: historical, overlord, middleManager, indexer, broker, coordinator, router.
+	// +optional
+	OrderOfUpgrade []string `json:"orderOfUpgrade,omitempty"`


Looks like the generated CRD manifests may still need to be included for these new API fields.

I see zz_generated.deepcopy.go was updated in the PR, but running make manifests generate locally still produced additional generated drift, especially in config/crd/bases/druid.apache.org_druids.yaml and chart/crds/druid.apache.org_druids.yaml.

Without those CRD updates, users installing from the manifests or Helm chart may not be able to persist orderOfUpgrade, orderOfUpgradeOfTiers, or tier as expected.

AdheipSingh · 2026-05-11T06:43:57Z

Thanks @aruraghuwanshi for the review. Will get back this week.

feat: tier-aware upgrade ordering for druid nodes

1d8da8c

AdheipSingh commented May 4, 2026

View reviewed changes

AdheipSingh mentioned this pull request May 6, 2026

fix: deterministic rollout order for multiple nodes per NodeType #15

Closed

4 tasks

aruraghuwanshi reviewed May 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/tier aware upgrade ordering#17

Feat/tier aware upgrade ordering#17
AdheipSingh wants to merge 1 commit into
apache:masterfrom
AdheipSingh:feat/tier-aware-upgrade-ordering

AdheipSingh commented May 4, 2026

Uh oh!

AdheipSingh commented May 4, 2026

Uh oh!

AdheipSingh May 4, 2026

Uh oh!

aruraghuwanshi May 9, 2026

Uh oh!

aruraghuwanshi May 9, 2026

Uh oh!

aruraghuwanshi May 10, 2026

Uh oh!

AdheipSingh commented May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

AdheipSingh commented May 4, 2026

Description

Key changed/added files in this PR

Uh oh!

AdheipSingh commented May 4, 2026

Note: The following is review is purely written by me and not edited or reviewed or taken any inspiration by any AI.

Problem Statement

Approach

Tiers

OrderOfUpgrade

OrderOfUpgradeOfTiers

Implementation

Uh oh!

AdheipSingh May 4, 2026

Choose a reason for hiding this comment

Uh oh!

aruraghuwanshi May 9, 2026

Choose a reason for hiding this comment

Uh oh!

aruraghuwanshi May 9, 2026

Choose a reason for hiding this comment

Uh oh!

aruraghuwanshi May 10, 2026

Choose a reason for hiding this comment

Uh oh!

AdheipSingh commented May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants