Skip to content

Add NodeCoordinator#68548

Merged
jason810496 merged 5 commits into
apache:mainfrom
shivaam:codex/typescript-coordinator-only
Jun 29, 2026
Merged

Add NodeCoordinator#68548
jason810496 merged 5 commits into
apache:mainfrom
shivaam:codex/typescript-coordinator-only

Conversation

@shivaam

@shivaam shivaam commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

This adds the Python-side NodeCoordinator needed to run TypeScript tasks through Airflow coordinator mode.

It pairs with the public TypeScript Task SDK interface in #67908. A Python Dag still defines the scheduling graph with stub tasks, while the task implementation can live in a TypeScript bundle. At execution time, Airflow routes the stub task to NodeCoordinator, which starts the Node.js bundle through the existing supervisor subprocess path.

The expected bundle layout is:

/opt/airflow/ts-bundles/
  bundle.mjs
  airflow-metadata.yaml

bundle.mjs is the Node.js entrypoint for the TypeScript task runtime. airflow-metadata.yaml describes the bundle and provides the supervisor schema version the coordinator should use when talking to the runtime.

Example metadata:

airflow_bundle_metadata_version: "1.0"
sdk:
  language: typescript
  version: "0.1.0-alpha.0"
  supervisor_schema_version: "2026-06-16"
source: src/airflow.ts
dags:
  sales_pipeline:
    tasks:
      - extract
      - transform

For this first PR, NodeCoordinator only needs sdk.supervisor_schema_version, but this metadata file follows the same general shape as Airflow executable bundle metadata so TypeScript bundle tooling can grow into it later.

Example Airflow config:

[sdk]
coordinators = {
    "ts": {
        "classpath": "airflow.sdk.coordinators.node.NodeCoordinator",
        "kwargs": {"bundles_root": ["/opt/airflow/ts-bundles"]}
    }
}
queue_to_coordinator = {"ts": "ts"}

This first version keeps bundle selection deliberately simple. bundles_root is an ordered fallback list of directories, and the first valid bundle.mjs with valid metadata is used. Dag/task-aware selection across multiple TypeScript bundles is left for follow-up work once the TypeScript bundle tooling exists.

Testing

I ran a full local daemon E2E with api-server, dag-processor, and scheduler to test this locally with my other code.

Follow-up work

  • Add the TypeScript SDK runtime PR with startCoordinator().
  • Add an official TypeScript pack/build tool that generates bundle.mjs and airflow-metadata.yaml.
  • Add Dag/task-aware multi-bundle selection after bundle tooling exists.
  • CI/CD tests

Was generative AI tooling used to co-author this PR?
  • Yes - OpenAI Codex

Generated-by: OpenAI Codex following the guidelines

@shivaam shivaam changed the title Add TypeScript task coordinator [Do not merge] Add TypeScript task coordinator Jun 15, 2026
@shivaam shivaam force-pushed the codex/typescript-coordinator-only branch from 694c6ef to 3b4654e Compare June 15, 2026 00:51
@shivaam shivaam marked this pull request as ready for review June 15, 2026 00:55
@uranusjr

Copy link
Copy Markdown
Member

I think the class name should be TypeScriptCoordinator since that’s the canonical spelling. The module name can probably just be ts but that’s not a big deal, either way works.

Comment thread task-sdk/src/airflow/sdk/coordinators/typescript/coordinator.py Outdated
@uranusjr

Copy link
Copy Markdown
Member

I’m thinking, maybe the coordinator should be NodeCoordinator instead. Go does not have a GoCoordinator but a more general ExecutableCoordinator. Java has JavaCoordinator, but that’s not (mainly) because the SDK targets Java (it doesn’t just target Java but all JVM languages), but more because the command that runs the SDK bundle is called java (even though the command runs a JVM—it’s what it is). So NodeCoordinator arguably is more suitable than TypeScriptCoordinator since the former better describes what the coordinator actually does, on the same level as the other existing coordinators.

Comment thread task-sdk/src/airflow/sdk/coordinators/typescript/coordinator.py Outdated

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Would like to discuss the high level opening question before diving into the actual implementation, thanks.

Comment thread task-sdk/src/airflow/sdk/coordinators/typescript/__init__.py Outdated
Comment thread task-sdk/src/airflow/sdk/coordinators/node/coordinator.py
@shivaam

shivaam commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

I’m thinking, maybe the coordinator should be NodeCoordinator instead. Go does not have a GoCoordinator but a more general ExecutableCoordinator. Java has JavaCoordinator, but that’s not (mainly) because the SDK targets Java (it doesn’t just target Java but all JVM languages), but more because the command that runs the SDK bundle is called java (even though the command runs a JVM—it’s what it is). So NodeCoordinator arguably is more suitable than TypeScriptCoordinator since the former better describes what the coordinator actually does, on the same level as the other existing coordinators.

Yes, that makes sense. I will change it.

@shivaam shivaam changed the title [Do not merge] Add TypeScript task coordinator Add TypeScript task coordinator Jun 28, 2026
@shivaam shivaam force-pushed the codex/typescript-coordinator-only branch from 3b4654e to 03f2d04 Compare June 28, 2026 19:29
@shivaam shivaam force-pushed the codex/typescript-coordinator-only branch from 03f2d04 to b106289 Compare June 28, 2026 19:33
Comment thread task-sdk/src/airflow/sdk/coordinators/node/coordinator.py

@uranusjr uranusjr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor request; should be good once suggestion is applied.

@jason810496 jason810496 changed the title Add TypeScript task coordinator Add NodeCoordinator Jun 29, 2026

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks.

@jason810496 jason810496 merged commit 9b6f6d3 into apache:main Jun 29, 2026
102 checks passed
karenbraganz pushed a commit to karenbraganz/airflow that referenced this pull request Jun 30, 2026
* Add TypeScript task coordinator

* Add newsfragment for TypeScript coordinator

* Rename TypeScript coordinator to Node coordinator

* Document Node coordinator usage for TypeScript SDK

* Improve Node coordinator metadata rejection logging
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants