Skip to content

feat(connectors): extend Postgres sink & source connectors, add integration tests#2579

Merged
spetz merged 5 commits intomasterfrom
postres_connector
Jan 19, 2026
Merged

feat(connectors): extend Postgres sink & source connectors, add integration tests#2579
spetz merged 5 commits intomasterfrom
postres_connector

Conversation

@spetz
Copy link
Copy Markdown
Contributor

@spetz spetz commented Jan 17, 2026

  • Extended Postgres sink/source connectors with JSON and raw message format support, multiple processing modes (mark/delete), and bytea column handling
  • Added comprehensive integration tests for sink and source connectors
  • Improved shared configuration loader with env overrides for array and strings
  • Updated Rust dependencies, while locking reqwest and a few others until fully compatible with other crates
  • Added optional verbose flag for sink and source configuration to toggle logging about processed messages

@spetz spetz added the connectors Connectors runtime label Jan 17, 2026
@spetz spetz force-pushed the postres_connector branch from f35753c to 8e9fe56 Compare January 17, 2026 21:42
@spetz spetz requested a review from hubcio January 17, 2026 21:43
@spetz spetz force-pushed the postres_connector branch from 8e9fe56 to 2fee83d Compare January 17, 2026 21:46
Copy link
Copy Markdown
Contributor

@hubcio hubcio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

besides that sink processes using batches but it inserts one row at a time, you can optimize, other than that looks good

@spetz
Copy link
Copy Markdown
Contributor Author

spetz commented Jan 19, 2026

besides that sink processes using batches but it inserts one row at a time, you can optimize, other than that looks good

Will fix, and also unify the configurations, to override what's necessary via existing env provider.

@spetz spetz merged commit 9319220 into master Jan 19, 2026
55 checks passed
@spetz spetz deleted the postres_connector branch January 19, 2026 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

connectors Connectors runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants