-
Notifications
You must be signed in to change notification settings - Fork 475
Scatter every-N-minutes schedules to reduce concurrent overlap #42524
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
2396b05
eade16d
d3b7890
4221eea
b08c92d
def9834
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # ADR-42524: Scatter Every-N-Minutes Schedules via Fuzzy Token Pipeline | ||
|
|
||
| **Date**: 2026-06-30 | ||
| **Status**: Draft | ||
| **Deciders**: Unknown | ||
|
|
||
| --- | ||
|
|
||
| ### Context | ||
|
|
||
| Workflows using "every N minutes" natural-language schedule syntax (e.g., `every 10 minutes`, `every 5m`) were compiled directly to standard cron `*/N * * * *` expressions. Because all workflows sharing the same interval produce the same cron pattern, they all fire simultaneously, creating predictable load spikes at minute boundaries. The codebase already contained a fuzzy-scatter pipeline (`ScatterSchedule`) that resolves intermediate `FUZZY:*` tokens to deterministically offset cron expressions for hourly and weekly schedules. Minute-interval schedules were not included in this pipeline, leaving a gap in the thundering-herd mitigation strategy. | ||
|
|
||
| ### Decision | ||
|
|
||
| We will extend the existing fuzzy-scatter pipeline to cover minute-interval schedules. The schedule parser now emits `FUZZY:EVERY_MINUTE/N * * * *` instead of `*/N * * * *` for "every N minutes" inputs. A new `handleEveryMinute` handler in the scatter layer resolves this token to `M/N * * * *`, where `M = stableHash(workflowIdentifier, N) ∈ [0, N-1]`, preserving the period while distributing start minutes across the clock face. The offset is deterministic per workflow identifier so recompilation produces the same cron without drift. Raw cron expressions (`*/N * * * *` typed directly by users) bypass scatter as before. | ||
|
|
||
| ### Alternatives Considered | ||
|
|
||
| #### Alternative 1: Keep `*/N * * * *` and accept simultaneous firing | ||
|
|
||
| The simplest option: do nothing. Workflows with the same interval continue to fire at the same minute. This was rejected because simultaneous firing creates measurable load spikes, and the infrastructure to mitigate it already exists for other schedule types — extending it to minute intervals is low-cost and consistent. | ||
|
|
||
| #### Alternative 2: Apply a random offset at compile time | ||
|
|
||
| Assign a random offset `M ∈ [0, N-1]` when compiling the schedule, storing the result as a plain cron expression. This avoids the intermediate `FUZZY:` token. It was rejected because a random offset changes on every recompilation, causing schedule drift — a workflow would silently shift its firing time whenever its definition is reprocessed. The deterministic hash approach used in the decision ensures the same workflow always produces the same cron, which is safer for audit trails and change detection. | ||
|
|
||
| #### Alternative 3: Offset by a fixed per-repo value rather than per-workflow hash | ||
|
|
||
| Use a single fixed offset derived from the repository rather than the workflow identifier. This was rejected because all workflows in the same repo would still cluster together at the same offset, merely shifting the load spike rather than distributing it. | ||
|
|
||
| ### Consequences | ||
|
|
||
| #### Positive | ||
| - Workflows sharing a minute interval are now spread across the clock face, reducing simultaneous firing and the associated load spikes. | ||
| - The offset is deterministic — same workflow identifier always produces the same cron, so recompilation is idempotent and auditable. | ||
| - The fix reuses the existing `stableHash` and `ScatterSchedule` infrastructure, keeping the implementation surface small and consistent with the hourly/weekly scatter approach. | ||
|
|
||
| #### Negative | ||
| - The compiled cron output changes from the intuitive `*/N` form to the less-familiar `M/N` step-with-start form. Both are semantically equivalent but the latter may surprise users reading raw cron values. | ||
| - Existing stored cron expressions (`*/N * * * *`) generated before this change are not retroactively updated; workflows will only receive a scattered offset on their next recompilation. | ||
|
|
||
| #### Neutral | ||
| - The intermediate `FUZZY:EVERY_MINUTE/N` token must now be understood by any tooling that inspects or stores cron expressions mid-pipeline. | ||
| - Tests for minute-interval schedules now assert offset-in-range (`[0, N-1]`) rather than exact cron strings, which is more robust to hash-function changes but loses exact value pinning (except in the cross-platform stability test). | ||
|
|
||
| --- | ||
|
|
||
| *ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.* |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -196,6 +196,7 @@ func ScatterSchedule(fuzzyCron, workflowIdentifier string) (string, error) { | |
| handleDaily, | ||
| handleHourlyWeekdays, | ||
| handleHourly, | ||
| handleEveryMinute, | ||
| handleWeeklyAround, | ||
| handleWeeklySpecific, | ||
| handleWeekly, | ||
|
|
@@ -325,6 +326,32 @@ func handleHourly(fuzzyCron, workflowIdentifier string) (string, bool, error) { | |
| return result, true, nil | ||
| } | ||
|
|
||
| // handleEveryMinute scatters "FUZZY:EVERY_MINUTE/N * * * *" to "M/N * * * *", | ||
| // where M is a deterministic offset in [0, N-1] derived from the workflow identifier. | ||
| // This ensures that concurrent "every N minutes" workflows start on different minutes, | ||
| // distributing execution across the clock face and reducing simultaneous load spikes. | ||
| func handleEveryMinute(fuzzyCron, workflowIdentifier string) (string, bool, error) { | ||
| const prefix = "FUZZY:EVERY_MINUTE/" | ||
| if !strings.HasPrefix(fuzzyCron, prefix) { | ||
| return "", false, nil | ||
| } | ||
| // parseHourlyInterval extracts the integer N from a "PREFIX/N ..." pattern; | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [/codebase-design] The inline comment explaining why @copilot please address this. |
||
| // it is reused here because the parsing logic is identical. | ||
| interval, err := parseHourlyInterval(fuzzyCron, prefix, "invalid fuzzy every-minute pattern", "invalid interval in fuzzy every-minute pattern") | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
💡 DetailsThe function name The comment already acknowledges this is a workaround ("parsing logic is identical"), which signals the abstraction is wrong rather than the reuse being safe. Suggested fix: Extract a generic |
||
| if err != nil { | ||
| return "", true, err | ||
| } | ||
| if interval < 1 { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [/diagnosing-bugs] The 💡 Why this matters
Add a test case with @copilot please address this. |
||
| return "", true, fmt.Errorf("invalid interval in fuzzy every-minute pattern: interval must be >= 1, got %d", interval) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Missing upper-bound validation for Only For Consider adding: if interval >= 60 {
return "", true, fmt.Errorf("invalid interval in fuzzy every-minute pattern: interval must be < 60, got %d (use 'every N hours' for larger intervals)", interval)
}@copilot please address this. |
||
| } | ||
| // stableHash returns a value in [0, interval-1], so offset is always a valid | ||
| // starting minute that preserves the N-minute period. | ||
| offset := stableHash(workflowIdentifier, interval) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bug: invalid cron minute field when The comment on lines 347–348 claims "offset is always a valid starting minute", but this is incorrect for Confirmed examples (FNV-1a hash, same algorithm as
Fix: cap the hash modulo at 60 so the offset always lands in offset := stableHash(workflowIdentifier, min(interval, 60))Or add an explicit upper-bound guard rejecting @copilot please address this. |
||
| result := fmt.Sprintf("%d/%d * * * *", offset, interval) | ||
| scheduleFuzzyScatterLog.Printf("FUZZY:EVERY_MINUTE/%d scattered: offset=%d, result=%s", interval, offset, result) | ||
| return result, true, nil | ||
| } | ||
|
|
||
| func handleWeeklyAround(fuzzyCron, workflowIdentifier string) (string, bool, error) { | ||
| const prefix = "FUZZY:WEEKLY_AROUND:" | ||
| if !strings.HasPrefix(fuzzyCron, prefix) { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -283,7 +283,7 @@ func formatShortDurationCron(interval int, unit string, hasWeekdaysSuffix bool) | |
| if hasWeekdaysSuffix { | ||
| return "", errors.New("minute intervals with 'on weekdays' are not supported") | ||
| } | ||
| return fmt.Sprintf("*/%d * * * *", interval), nil | ||
| return fmt.Sprintf("FUZZY:EVERY_MINUTE/%d * * * *", interval), nil | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Empty 💡 Details and suggested fixBefore this PR, // schedule_preprocessing.go:46
if parser.IsFuzzyCron(parsedCron) && c.workflowIdentifier != "" {
// scatter happens
}
// else: raw FUZZY token is written to the output YAML unchangedAny caller that invokes the parser without Note: Suggested fix: Emit the FUZZY token from the parser as today, but inside |
||
| case "h": | ||
|
Comment on lines
283
to
287
|
||
| return formatHourlyIntervalCron(interval, hasWeekdaysSuffix), nil | ||
| case "d": | ||
|
|
@@ -303,7 +303,7 @@ func formatLongIntervalCron(interval int, unit string, hasWeekdaysSuffix bool) ( | |
| if hasWeekdaysSuffix { | ||
| return "", errors.New("minute intervals with 'on weekdays' are not supported") | ||
| } | ||
| return fmt.Sprintf("*/%d * * * *", interval), nil | ||
| return fmt.Sprintf("FUZZY:EVERY_MINUTE/%d * * * *", interval), nil | ||
| case "hours": | ||
| return formatHourlyIntervalCron(interval, hasWeekdaysSuffix), nil | ||
| case "days": | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -101,6 +101,36 @@ func TestScatterScheduleCrossPlatformConsistency(t *testing.T) { | |
| workflowIdentifier: "workflow-a.md", | ||
| expectedCron: "21 8 * * 1", | ||
| }, | ||
| { | ||
| name: "every 10 minutes - workflow-a.md", | ||
| fuzzyCron: "FUZZY:EVERY_MINUTE/10 * * * *", | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [/tdd] Missing stability pin for 💡 Suggested addition{
name: "every 1 minute - any workflow (no scatter, mod-1 always 0)",
fuzzyCron: "FUZZY:EVERY_MINUTE/1 * * * *",
workflowIdentifier: "workflow-a.md",
expectedCron: "0/1 * * * *", // stableHash(_, 1) == 0 for all identifiers
},A comment in the test body explaining why this always yields @copilot please address this. |
||
| workflowIdentifier: "workflow-a.md", | ||
| expectedCron: "1/10 * * * *", | ||
| }, | ||
| { | ||
| name: "every 10 minutes - workflow-b.md", | ||
| fuzzyCron: "FUZZY:EVERY_MINUTE/10 * * * *", | ||
| workflowIdentifier: "workflow-b.md", | ||
| expectedCron: "2/10 * * * *", | ||
| }, | ||
| { | ||
| name: "every 5 minutes - workflow-a.md", | ||
| fuzzyCron: "FUZZY:EVERY_MINUTE/5 * * * *", | ||
| workflowIdentifier: "workflow-a.md", | ||
| expectedCron: "1/5 * * * *", | ||
| }, | ||
| { | ||
| name: "every 30 minutes - workflow-a.md", | ||
| fuzzyCron: "FUZZY:EVERY_MINUTE/30 * * * *", | ||
| workflowIdentifier: "workflow-a.md", | ||
| expectedCron: "21/30 * * * *", | ||
| }, | ||
| { | ||
| name: "every 15 minutes - workflow-b.md", | ||
| fuzzyCron: "FUZZY:EVERY_MINUTE/15 * * * *", | ||
| workflowIdentifier: "workflow-b.md", | ||
| expectedCron: "7/15 * * * *", | ||
| }, | ||
| } | ||
|
|
||
| for _, tt := range tests { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,6 +4,7 @@ package workflow | |
|
|
||
| import ( | ||
| "fmt" | ||
| "strconv" | ||
| "strings" | ||
| "testing" | ||
| ) | ||
|
|
@@ -216,7 +217,7 @@ func TestSchedulePreprocessingShorthandOnString(t *testing.T) { | |
| frontmatter: map[string]any{ | ||
| "on": "every 10 minutes", | ||
| }, | ||
| expectedCron: "*/10 * * * *", | ||
| checkScattered: true, // Scattered minute-interval schedule | ||
| expectWorkflowDispatch: true, | ||
| }, | ||
| { | ||
|
|
@@ -342,6 +343,20 @@ func TestSchedulePreprocessingShorthandOnString(t *testing.T) { | |
| fields := strings.Fields(actualCron) | ||
| if len(fields) != 5 { | ||
| t.Errorf("expected 5 fields in cron expression, got %d: %s", len(fields), actualCron) | ||
| return | ||
| } | ||
| // If the minute field uses M/N start/step syntax, verify that the | ||
| // offset is a valid integer in [0, N-1]. A non-integer offset (e.g. | ||
| // "*" from an unscattered "*/N") is itself a failure. | ||
| minuteField := fields[0] | ||
| if parts := strings.SplitN(minuteField, "/", 2); len(parts) == 2 { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [/tdd] The offset bounds-check block (lines 348–356) is duplicated verbatim in 💡 Suggested helper// assertValidScatteredMinuteCron verifies that actualCron is a valid 5-field
// cron expression whose minute field uses M/N syntax with M in [0, N-1].
func assertValidScatteredMinuteCron(t *testing.T, actualCron string) {
t.Helper()
if strings.HasPrefix(actualCron, "FUZZY:") {
t.Errorf("expected scattered cron, got fuzzy: %s", actualCron)
return
}
fields := strings.Fields(actualCron)
if len(fields) != 5 {
t.Errorf("expected 5 fields in cron expression, got %d: %s", len(fields), actualCron)
return
}
if parts := strings.SplitN(fields[0], "/", 2); len(parts) == 2 {
offset, offsetErr := strconv.Atoi(parts[0])
interval, intervalErr := strconv.Atoi(parts[1])
if offsetErr == nil && intervalErr == nil && (offset < 0 || offset >= interval) {
t.Errorf("offset %d is not in [0, %d): %s", offset, interval, actualCron)
}
}
t.Logf("Successfully scattered schedule to: %s", actualCron)
}Then replace both @copilot please address this. |
||
| offset, offsetErr := strconv.Atoi(parts[0]) | ||
| interval, intervalErr := strconv.Atoi(parts[1]) | ||
| if offsetErr != nil || intervalErr != nil { | ||
| t.Errorf("minute field %q is not a valid M/N expression: %s", minuteField, actualCron) | ||
| } else if offset < 0 || offset >= interval { | ||
| t.Errorf("offset %d is not in [0, %d): %s", offset, interval, actualCron) | ||
| } | ||
| } | ||
| t.Logf("Successfully scattered schedule to: %s", actualCron) | ||
| } else if tt.expectedCron != "" { | ||
|
|
@@ -355,11 +370,13 @@ func TestSchedulePreprocessingShorthandOnString(t *testing.T) { | |
|
|
||
| func TestSchedulePreprocessing(t *testing.T) { | ||
| tests := []struct { | ||
| name string | ||
| frontmatter map[string]any | ||
| expectedCron string | ||
| expectedError bool | ||
| errorSubstring string | ||
| name string | ||
| frontmatter map[string]any | ||
| workflowIdentifier string | ||
| expectedCron string | ||
| checkScattered bool | ||
| expectedError bool | ||
| errorSubstring string | ||
| }{ | ||
| { | ||
| name: "daily schedule", | ||
|
|
@@ -398,7 +415,8 @@ func TestSchedulePreprocessing(t *testing.T) { | |
| }, | ||
| }, | ||
| }, | ||
| expectedCron: "*/10 * * * *", | ||
| workflowIdentifier: "test-workflow.md", | ||
| checkScattered: true, | ||
| }, | ||
| { | ||
| name: "existing cron expression unchanged", | ||
|
|
@@ -469,7 +487,8 @@ func TestSchedulePreprocessing(t *testing.T) { | |
| "schedule": "every 10 minutes", | ||
| }, | ||
| }, | ||
| expectedCron: "*/10 * * * *", | ||
| workflowIdentifier: "test-workflow.md", | ||
| checkScattered: true, | ||
| }, | ||
| { | ||
| name: "shorthand string format - existing cron", | ||
|
|
@@ -495,6 +514,9 @@ func TestSchedulePreprocessing(t *testing.T) { | |
| for _, tt := range tests { | ||
| t.Run(tt.name, func(t *testing.T) { | ||
| compiler := NewCompiler() | ||
| if tt.workflowIdentifier != "" { | ||
| compiler.SetWorkflowIdentifier(tt.workflowIdentifier) | ||
| } | ||
| err := compiler.preprocessScheduleFields(tt.frontmatter, "", "") | ||
|
|
||
| if tt.expectedError { | ||
|
|
@@ -519,8 +541,35 @@ func TestSchedulePreprocessing(t *testing.T) { | |
| firstSchedule := scheduleArray[0].(map[string]any) | ||
| actualCron := firstSchedule["cron"].(string) | ||
|
|
||
| if actualCron != tt.expectedCron { | ||
| t.Errorf("expected cron '%s', got '%s'", tt.expectedCron, actualCron) | ||
| if tt.checkScattered { | ||
| // Should be scattered to a valid cron (not fuzzy) | ||
| if strings.HasPrefix(actualCron, "FUZZY:") { | ||
| t.Errorf("expected scattered cron, got fuzzy: %s", actualCron) | ||
| } | ||
| fields := strings.Fields(actualCron) | ||
| if len(fields) != 5 { | ||
| t.Errorf("expected 5 fields in cron expression, got %d: %s", len(fields), actualCron) | ||
| return | ||
| } | ||
| // If the minute field uses the M/N start/step syntax, verify | ||
| // that the offset is a valid integer in [0, N-1] so the period | ||
| // is preserved. A non-integer offset (e.g. "*" from an | ||
| // unscattered "*/N") is itself a failure. | ||
| minuteField := fields[0] | ||
| if parts := strings.SplitN(minuteField, "/", 2); len(parts) == 2 { | ||
| offset, offsetErr := strconv.Atoi(parts[0]) | ||
| interval, intervalErr := strconv.Atoi(parts[1]) | ||
| if offsetErr != nil || intervalErr != nil { | ||
| t.Errorf("minute field %q is not a valid M/N expression: %s", minuteField, actualCron) | ||
| } else if offset < 0 || offset >= interval { | ||
| t.Errorf("offset %d is not in [0, %d): %s", offset, interval, actualCron) | ||
| } | ||
| } | ||
| t.Logf("Successfully scattered schedule to: %s", actualCron) | ||
| } else if tt.expectedCron != "" { | ||
| if actualCron != tt.expectedCron { | ||
| t.Errorf("expected cron '%s', got '%s'", tt.expectedCron, actualCron) | ||
| } | ||
| } | ||
| }) | ||
| } | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[/tdd] No dedicated
TestScatterScheduleEveryMinutewas added toschedule_fuzzy_scatter_test.go, yet every other handler (e.g.TestScatterScheduleHourly) has its own test function that verifies the result is a valid cron expression and checks the output structure.💡 Suggested test skeleton
@copilot please address this.