Fix node options and OOM retries

[nicktrn]

We needed to handle underscores in the flags and prevent duplicate runner names when retrying on larger machines. The retry span improvements have been forward ported from v3.

Summary by CodeRabbit

• New Features

  • Enhanced runner identification by incorporating attempt details for improved handling of retries.
  • Improved run retry event logging by dynamically including contextual machine information.

• Refactor

  • Updated production build configurations by refining memory settings.

• Tests

  • Expanded test coverage to verify consistent processing of command-line flags by standardizing formats (underscores to hyphens).

[changeset-bot]

⚠️ No Changeset found

Latest commit: ae263b3

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Walkthrough

This update enhances the runner identification and event reporting mechanisms. The getRunnerId function now accepts an optional attempt parameter to include attempt details. This change propagates to Docker and Kubernetes workload managers by passing the new parameter when invoking the function. Additionally, event handling for run retries has been refined by adding a dynamic message and an extra attribute indicating the next machine after an OOM event. Other updates include a Docker image configuration change and deduplication logic improvements for command-line flags.

Changes

File(s)	Changes Summary
apps/supervisor/src/util.ts	Modified getRunnerId to accept an optional attemptNumber parameter and append "attempt-{attemptNumber}" when applicable.
apps/supervisor/src/workloadManager/docker.ts apps/supervisor/src/workloadManager/kubernetes.ts	Updated create methods to pass the new attemptNumber parameter (opts.nextAttemptNumber) to getRunnerId.
apps/webapp/app/v3/runEngineHandlers.server.ts	Revised the "runRetryScheduled" event handling by constructing a dynamic retryMessage and adding a new event attribute nextMachine based on run.nextMachineAfterOOM.
internal-packages/run-engine/src/engine/eventBus.ts internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts	Added a new optional property nextMachineAfterOOM to the runRetryScheduled event type and included it in the emitted event payload.
packages/cli-v3/src/deploy/buildImage.ts	Removed the NODE_OPTIONS="--max_old_space_size=8192" environment variable from the Dockerfile configuration.
packages/core/src/v3/build/flags.ts packages/core/src/v3/build/flags.test.ts	Updated dedupFlags logic to replace underscores with hyphens in flag keys and added tests to verify this behavior.

Sequence Diagram(s)

sequenceDiagram
    participant WM as WorkloadManager (Docker/K8s)
    participant U as Util (getRunnerId)
    
    WM->>U: Call getRunnerId(runId, attemptNumber)
    U-->>WM: Return formatted runner id ("runner-{runId}-attempt-{attemptNumber}?")

Loading

sequenceDiagram
    participant RA as RunAttemptSystem
    participant EB as EventBus
    
    RA->>EB: Emit runRetryScheduled event with retryMessage & nextMachineAfterOOM
    EB-->>RA: Acknowledge event processing

Loading

Possibly related PRs

• Prevent stuck runs after duplicate dependency resume messages #1459: Updates to getRunnerId across workload managers align directly with the changes here.
• fix metadata system not updating because of dual package hazard #1428: Similar modifications to the function invocation in different contexts, reinforcing this update.
• Fix v4 restore race condition #1870: Reflects consistent changes in runner ID generation for Docker and Kubernetes, matching this PR’s approach.

Suggested reviewers

• matt-aitken

Poem

Hopping through lines of code so bright,
I found new tweaks that feel just right.
Runner IDs now sing a fresher tune,
With retry events under a new moon.
From Docker to K8s, the changes gleam—
This rabbit cheers, coding like a dream!
🐇✨ Happy hops and bug-free streams!

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a51b9b9 and 98749b4.

📒 Files selected for processing (9)

• apps/supervisor/src/util.ts (1 hunks)
• apps/supervisor/src/workloadManager/docker.ts (1 hunks)
• apps/supervisor/src/workloadManager/kubernetes.ts (1 hunks)
• apps/webapp/app/v3/runEngineHandlers.server.ts (1 hunks)
• internal-packages/run-engine/src/engine/eventBus.ts (1 hunks)
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts (1 hunks)
• packages/cli-v3/src/deploy/buildImage.ts (1 hunks)
• packages/core/src/v3/build/flags.test.ts (1 hunks)
• packages/core/src/v3/build/flags.ts (1 hunks)

🧰 Additional context used

🧬 Code Definitions (3)

apps/supervisor/src/workloadManager/kubernetes.ts (1)

apps/supervisor/src/util.ts (1)

• getRunnerId (8-16)

packages/core/src/v3/build/flags.test.ts (1)

packages/core/src/v3/build/flags.ts (1)

• dedupFlags (20-47)

apps/webapp/app/v3/runEngineHandlers.server.ts (1)

apps/webapp/app/v3/eventRepository.server.ts (1)

• eventRepository (1264-1264)

⏰ Context from checks skipped due to timeout of 90000ms (6)

• GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
• GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
• GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
• GitHub Check: units / 🧪 Unit Tests
• GitHub Check: typecheck / typecheck
• GitHub Check: Analyze (javascript-typescript)

🔇 Additional comments (11)

packages/core/src/v3/build/flags.test.ts (1)

21-24: Great addition of test cases for underscore handling in flags.

These test cases effectively verify that the dedupFlags function treats underscores as hyphens, which is an important edge case to cover. The test ensures both the simple case (--debug_level=info → --debug-level=info) and the deduplication case (--debug_level=info --debug-level=warn → --debug-level=warn) work correctly.

packages/core/src/v3/build/flags.ts (2)

30-30: Good implementation of underscore replacement for flags with values.

The modification properly handles the conversion of underscores to hyphens in flag names, which aligns with common CLI flag conventions and the PR's objective to fix handling of underscores in flags.

---

34-34: Correctly implemented underscore replacement for flags without values.

The change ensures consistent handling of underscores across all flag types, whether they have associated values or not. This creates a uniform approach to flag normalization throughout the codebase.

internal-packages/run-engine/src/engine/eventBus.ts (1)

88-88: Good addition to track next machine after OOM events.

This new optional property enhances the event system by allowing it to track which machine should be used after an out-of-memory event occurs. This will improve retry handling when jobs fail due to memory constraints.

packages/cli-v3/src/deploy/buildImage.ts (1)

691-691: Good removal of hard-coded Node memory limit.

Removing the NODE_OPTIONS="--max_old_space_size=8192" environment variable is a positive change. This allows Node.js processes to use the container-level memory limits rather than being artificially constrained, which can help address out-of-memory issues mentioned in the PR description.

apps/supervisor/src/workloadManager/kubernetes.ts (1)

34-34: Good enhancement to prevent duplicate runner IDs.

Adding the attempt number to the runner ID generation process ensures unique identification for each retry attempt. This aligns with the PR objective of "preventing duplicate runner names during retries on larger machines".

internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts (1)

695-695: Good implementation of machine tracking for OOM retries.

Adding the nextMachineAfterOOM property when emitting the runRetryScheduled event properly implements the type changes from the EventBus. This allows the system to track which machine should be used for the next attempt after an out-of-memory failure, supporting better resource allocation during retries.

apps/supervisor/src/workloadManager/docker.ts (1)

25-25: Update enhances runner ID generation for retry attempts.

The code now passes the nextAttemptNumber parameter to the getRunnerId function, which will create unique container names for retry attempts. This change helps prevent name collisions when containers are retried on the same machine.

apps/supervisor/src/util.ts (1)

8-15: Good implementation of attempt-specific runner IDs.

The updated getRunnerId function now intelligently incorporates the attempt number into the runner ID, but only when it's greater than 1. This approach maintains backward compatibility while solving the container name collision issue during retries.

Benefits:

• Prevents duplicate runner names during retries
• Maintains the original format for first attempts
• Improves traceability by making the attempt number explicit in logs

apps/webapp/app/v3/runEngineHandlers.server.ts (2)

329-335: Enhanced retry messaging with OOM context.

The dynamic message construction now includes specific information about OOM (Out of Memory) events, improving observability during debugging sessions.

---

341-341: Added machine information for OOM retries.

Adding the nextMachine attribute to the event record provides valuable context about which machine will be used for the retry after an OOM event. This helps with tracking the progression of retries across different machine types.

✨ Finishing Touches

• 📝 Generate Docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai plan to trigger planning for file edits and PR creation.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.