Skip to content

feat(cdk): connector builder support for file uploader#492

Closed
Aldo Gonzalez (aldogonzalez8) wants to merge 19 commits into
mainfrom
ac8/file-api/connector-builder-support-2
Closed

feat(cdk): connector builder support for file uploader#492
Aldo Gonzalez (aldogonzalez8) wants to merge 19 commits into
mainfrom
ac8/file-api/connector-builder-support-2

Conversation

@aldogonzalez8
Copy link
Copy Markdown
Contributor

@aldogonzalez8 Aldo Gonzalez (aldogonzalez8) commented Apr 17, 2025 •
edited
Loading

What

We want to implement support for the connector builder to communicate about the files that will be uploaded for the streams.

How

Current Implementation:

  • Runtime (emit_connector_builder_messages=False):
    • FileUploader is instantiated with a standard FileWriter, which performs the actual file writing.
  • Connector Builder (emit_connector_builder_messages=True):
    • FileUploader is instantiated with a NoopFileWriter, preventing actual file writes.
    • This FileUploader instance is then wrapped by ConnectorBuilderFileUploader.
    • ConnectorBuilderFileUploader.upload() first calls the wrapped FileUploader.upload() (which populates record.file_reference but doesn't write the file) and then performs the Connector Builder-specific action of copying file_reference attributes into record.data.

maxi297 and others added 12 commits March 19, 2025 12:16
PoC for file upload
d1d5107
Auto-fix lint and format issues
de09ae8
fix mypy
b410187
@aldogonzalez8 @maxi297
PoC for emit file reference record (#443)
b122c0a 
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: octavia-squidington-iii <contact@airbyte.com>
@aldogonzalez8
merge from main
4c29c91
@aldogonzalez8 @maxi297
@aaronsteers @devin-ai-integration
feat(PoC): adjust file-based and file uploader component to latest pr…
b5b8197 
…otocol changes. (#457)

Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: octavia-squidington-iii <contact@airbyte.com>
Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@aldogonzalez8
merge from main
d060a8b
@aldogonzalez8
poetry lock
87c20b6
@aldogonzalez8
file-api: add checks for missin values/paths
ab5d4e7
@aldogonzalez8
file-api: fix unit tests
5242876
Auto-fix lint and format issues
8f14512
@aldogonzalez8
connector builder: initial changes to pass file reference info to data
3c0c0ab
@github-actions github-actions Bot added the enhancement New feature or request label Apr 17, 2025
@aldogonzalez8
Copy link
Copy Markdown
Contributor Author

Aldo Gonzalez (aldogonzalez8) commented Apr 17, 2025 •
edited by github-actions Bot
Loading

/autofix

Auto-Fix Job Info

This job attempts to auto-fix any linting or formating issues. If any fixes are made,
those changes will be automatically committed and pushed back to the PR.

Note: This job can only be run by maintainers. On PRs from forks, this command requires
that the PR author has enabled the Allow edits from maintainers option.

PR auto-fix job started... Check job output.

✅ Changes applied successfully.

@aldogonzalez8
Copy link
Copy Markdown
Contributor Author

Aldo Gonzalez (aldogonzalez8) commented Apr 17, 2025 •
edited by github-actions Bot
Loading

/autofix

Auto-Fix Job Info

This job attempts to auto-fix any linting or formating issues. If any fixes are made,
those changes will be automatically committed and pushed back to the PR.

Note: This job can only be run by maintainers. On PRs from forks, this command requires
that the PR author has enabled the Allow edits from maintainers option.

PR auto-fix job started... Check job output.

✅ Changes applied successfully.

@aldogonzalez8
Copy link
Copy Markdown
Contributor Author

Aldo Gonzalez (aldogonzalez8) commented Apr 17, 2025 •
edited by github-actions Bot
Loading

/autofix

Auto-Fix Job Info

This job attempts to auto-fix any linting or formating issues. If any fixes are made,
those changes will be automatically committed and pushed back to the PR.

Note: This job can only be run by maintainers. On PRs from forks, this command requires
that the PR author has enabled the Allow edits from maintainers option.

PR auto-fix job started... Check job output.

🟦 Job completed successfully (no changes).

@aldogonzalez8
Copy link
Copy Markdown
Contributor Author

Aldo Gonzalez (aldogonzalez8) commented Apr 18, 2025 •
edited by github-actions Bot
Loading

/autofix

Auto-Fix Job Info

This job attempts to auto-fix any linting or formating issues. If any fixes are made,
those changes will be automatically committed and pushed back to the PR.

Note: This job can only be run by maintainers. On PRs from forks, this command requires
that the PR author has enabled the Allow edits from maintainers option.

PR auto-fix job started... Check job output.

✅ Changes applied successfully.

@aldogonzalez8 Aldo Gonzalez (aldogonzalez8) changed the title feat(connector builder): initial changes to pass file reference info to data feat(cdk): connector builder support for file uploader Apr 18, 2025
@aldogonzalez8 Aldo Gonzalez (aldogonzalez8) marked this pull request as ready for review April 18, 2025 15:44
Base automatically changed from maxi297/poc-file-upload to main April 22, 2025 14:41
@aldogonzalez8
Copy link
Copy Markdown
Contributor Author

Aldo Gonzalez (aldogonzalez8) commented Apr 23, 2025

Closed in favor of #503

@aldogonzalez8
Copy link
Copy Markdown
Contributor Author

Aldo Gonzalez (aldogonzalez8) commented Apr 23, 2025

Closed in favor of #503

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maxi297 Maxime Carbonneau-Leclerc (maxi297) Awaiting requested review from maxi297

Assignees

@aldogonzalez8 Aldo Gonzalez (aldogonzalez8)

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@aldogonzalez8