Skip to content

Add TypeScript coordinator runtime#69079

Merged
jason810496 merged 9 commits into
apache:mainfrom
shivaam:feat/add-ts-sdk-coordinator-runtime
Jul 3, 2026
Merged

Add TypeScript coordinator runtime#69079
jason810496 merged 9 commits into
apache:mainfrom
shivaam:feat/add-ts-sdk-coordinator-runtime

Conversation

@shivaam

@shivaam shivaam commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

Adds the TypeScript coordinator runtime to @apache-airflow/ts-sdk, the
follow-up to #67908 (TypeScript task SDK public interface). This lets Airflow
execute registered TypeScript task handlers through a bundled Node.js entrypoint.

  • Adds the coordinator runtime that speaks the supervisor wire protocol
    (length-prefixed msgpack frames) over the --comm and --logs sockets:
    connects, receives the task startup message, dispatches to the registered
    handler for the Dag/task pair, and reports the terminal task state.
  • Adds TaskClient-backed task-time data access: Variables (get / getOrThrow),
    XCom (get / set, including the automatic return_value push for non-undefined
    handler returns), and Connections.
  • Adds cooperative cancellation via ctx.signal (AbortSignal); SIGTERM/SIGINT
    abort the signal with a grace period before force-exit.
  • Adds generated supervisor wire types (src/generated/supervisor.ts) plus a
    generate:supervisor script, and a ./coordinator package export subpath.
  • Adds an example/ showing a Python stub Dag + TypeScript handlers, bundled
    with esbuild, and documents the bundle layout and Airflow coordinator config.

This is intentionally still Python-stub-Dag mode: TypeScript Dag declaration is
not introduced here. TypeScript registers handlers for Dag/task IDs declared by
Python stub tasks. The example bundles with esbuild and writes
airflow-metadata.yaml manually; folding metadata generation into the build
script is left to a separate follow-up.

Testing performed:

  • Unit tests and full e2e test with the airflow coordinator. Readme to run tests is available under examples.

Next steps

  • Combine airflow-metadata.yml with the .mjs file and introduce a ts pack tool.

@shivaam shivaam force-pushed the feat/add-ts-sdk-coordinator-runtime branch 3 times, most recently from 503a0e1 to cb139b3 Compare June 28, 2026 23:43
@shivaam shivaam marked this pull request as ready for review June 29, 2026 00:06
@shivaam

shivaam commented Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

@uranusjr @jason810496 PR for TS sdk runtime.

shivaam added 5 commits June 28, 2026 21:16
The runtime should keep generated supervisor payloads as the source of truth while making the discriminators required where the coordinator constructs or narrows wire messages.
@shivaam shivaam force-pushed the feat/add-ts-sdk-coordinator-runtime branch from cb139b3 to 5de8276 Compare June 29, 2026 04:17
Comment thread ts-sdk/src/coordinator/comm-channel.ts Outdated
Comment thread ts-sdk/src/coordinator/runtime.ts
Comment thread ts-sdk/src/generated/supervisor.ts
Comment thread ts-sdk/src/coordinator/log-channel.ts
@uranusjr

Copy link
Copy Markdown
Member

Need to fix the static check failure obviously but otherwise very nice work

@shivaam

shivaam commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

Need to fix the static check failure obviously but otherwise very nice work

Thank you. I have been spending the last couple of months understanding the Task SDK execution path, edge worker work, and coordinator model, so it’s really encouraging to see this coming together.

I’ve pushed a follow-up for the static check failure.

@jason810496 jason810496 self-requested a review July 1, 2026 03:51

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! LGTM overall, thanks.
Btw, does it mean that with this PR, the TS-SDK "works" (at least able to execute the Multi-Lang Dag e2e in breeze)?
I didn't test it manually.

Comment thread ts-sdk/src/generated/supervisor.ts
Comment thread ts-sdk/tests/coordinator/log-channel.test.ts
Comment thread ts-sdk/src/coordinator/client.ts
Comment thread ts-sdk/src/coordinator/client.ts
Comment thread ts-sdk/src/coordinator/log-channel.ts
@shivaam

shivaam commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Nice! LGTM overall, thanks. Btw, does it mean that with this PR, the TS-SDK "works" (at least able to execute the Multi-Lang Dag e2e in breeze)? I didn't test it manually.

Yes, the multi-lang dag will work after this PR. Just the bundler and packer needs to be implemented so you will have to create airflow-metadata.yml file. The readme in the example describes how to run it

@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 2, 2026

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Here're all the follow-up issues that I saw during the review so far. I prefer to have real issues to keep track of them instead of having follow-up statements in the review comments that might be missed in the long run.

  • #69286 - TypeScript SDK: add a ts pack bundle build tool
  • #69287 - TypeScript SDK: Dag/task-aware multi-bundle selection in NodeCoordinator
  • #69288 - TypeScript SDK: native TypeScript Dag declaration (remove Python stub requirement)
  • #69289 - TypeScript SDK: coordinator schema-mapping tests (camelCase to snake_case)
  • #69290 - TypeScript SDK: Airflow-level integration tests for coordinator-mode tasks
  • #69291 - TypeScript SDK: refactor shared logic between NodeCoordinator and ExecutableCoordinator
  • #69292 - TypeScript SDK: log socket buffering and reconnect hardening

@jason810496 jason810496 merged commit cd2e3ae into apache:main Jul 3, 2026
78 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants