From e1802667e784f778952f58002e2f56cd3675ada2 Mon Sep 17 00:00:00 2001 From: "Aaron (\"AJ\") Steers" Date: Mon, 9 Dec 2024 10:11:22 -0800 Subject: [PATCH 1/5] docs: add SDM release troubleshooting info --- docs/RELEASES.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/RELEASES.md b/docs/RELEASES.md index 5a92e090b..a2fba5b48 100644 --- a/docs/RELEASES.md +++ b/docs/RELEASES.md @@ -26,3 +26,25 @@ This process is slightly different from the above, since we don't necessarily wa 3. Select `main` or your dev branch from the "Use workflow from" dropdown. 4. Select your options and click "Run workflow". 5. Monitor the workflow to ensure the process has succeeded. + +## Understanding and Debugging Builder and SDM Releases + +### How Connector Builder uses SDM/CDK + +The Connector Builder (written in Java) calls the CDK Python package directly, executing the CDK's Source Declarative Manfiest code via Python processes on the Builder container. (The Connector Builder does not directly invoke the SDM image, but there is an open project to change this in the future.) + +Our publish flow sends a PR to the Builder repo (`airbyte-platform-internal`) to bump the version used in Builder. The Marketplace Contributions team (aka Connector Builder maintainers) will review and merge the PR. + +### How the SDM Image is used in Platform + +The platform scans DockerHub on specific intervals, and has logic to periodically bump the used version based on semver rules on the DockerTag. TODO: Get more info this. + +Note: Currently we don't pre-test images in Platform so manual testing is needed. + +### How to confirm what SDM version is used on the Platform + +TODO: ... + +### How to pretest changes to SDM images manually + +TODO: ... From 61a405d3e3b4c0d86a6cc63da3c4ca27e6ef1e1c Mon Sep 17 00:00:00 2001 From: "Aaron (\"AJ\") Steers" Date: Mon, 9 Dec 2024 11:13:24 -0800 Subject: [PATCH 2/5] Apply suggestions from code review --- docs/RELEASES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/RELEASES.md b/docs/RELEASES.md index a2fba5b48..170dbed9a 100644 --- a/docs/RELEASES.md +++ b/docs/RELEASES.md @@ -37,7 +37,7 @@ Our publish flow sends a PR to the Builder repo (`airbyte-platform-internal`) to ### How the SDM Image is used in Platform -The platform scans DockerHub on specific intervals, and has logic to periodically bump the used version based on semver rules on the DockerTag. TODO: Get more info this. +The platform scans DockerHub at an [every 10 minutes cadence](https://github.com/airbytehq/airbyte-platform-internal/blob/d744174c0f3ca8fa70f3e05cca6728f067219752/oss/airbyte-cron/src/main/java/io/airbyte/cron/jobs/DeclarativeSourcesUpdater.java) as of 2024-12-09. Based on that DockerHub scan, the platform bumps the default SDM version that is stored in the `declarative_manifest_image_version` table in prod. Note: Currently we don't pre-test images in Platform so manual testing is needed. From 1e0d5fda36661199a6e810728847e8987389b2ed Mon Sep 17 00:00:00 2001 From: "Aaron (\"AJ\") Steers" Date: Mon, 9 Dec 2024 11:37:36 -0800 Subject: [PATCH 3/5] Apply suggestions from code review --- docs/RELEASES.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/RELEASES.md b/docs/RELEASES.md index 170dbed9a..ed64a5fea 100644 --- a/docs/RELEASES.md +++ b/docs/RELEASES.md @@ -43,7 +43,13 @@ Note: Currently we don't pre-test images in Platform so manual testing is needed ### How to confirm what SDM version is used on the Platform -TODO: ... +Currently there are two ways to do this. + +The first option is to look in the `declarative_manifest_image_version` database table in Prod. + +If that is not available as an option, you can run an Builder-created connector in Cloud and note the version number printed in the logs. Warning: this may not be indicative if that connector instance has been manually pinned to a specific version. + +TODO: Would be great to find a way to inspect directly without requiring direct prod DB access. ### How to pretest changes to SDM images manually From dc47cac3c0a367fc631854c0c12084f49ac24921 Mon Sep 17 00:00:00 2001 From: "Aaron (\"AJ\") Steers" Date: Mon, 9 Dec 2024 11:39:26 -0800 Subject: [PATCH 4/5] Update docs/RELEASES.md --- docs/RELEASES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/RELEASES.md b/docs/RELEASES.md index ed64a5fea..fca045700 100644 --- a/docs/RELEASES.md +++ b/docs/RELEASES.md @@ -45,7 +45,7 @@ Note: Currently we don't pre-test images in Platform so manual testing is needed Currently there are two ways to do this. -The first option is to look in the `declarative_manifest_image_version` database table in Prod. +The first option is to look in the `declarative_manifest_image_version` database table in Prod. If that is not available as an option, you can run an Builder-created connector in Cloud and note the version number printed in the logs. Warning: this may not be indicative if that connector instance has been manually pinned to a specific version. From 2dc486c96567c305819f17b44d9f8272dc30af7a Mon Sep 17 00:00:00 2001 From: "Aaron (\"AJ\") Steers" Date: Mon, 9 Dec 2024 13:41:31 -0800 Subject: [PATCH 5/5] apply new content Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com> --- docs/RELEASES.md | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/RELEASES.md b/docs/RELEASES.md index fca045700..cf74d4232 100644 --- a/docs/RELEASES.md +++ b/docs/RELEASES.md @@ -53,4 +53,30 @@ TODO: Would be great to find a way to inspect directly without requiring direct ### How to pretest changes to SDM images manually -TODO: ... +To manually test changes against a dev image of SDM before committing to a release, first use the Publishing & Packaging workflow to publish a pre-release version of the CDK/SDM. Be sure to uncheck the option to create a connector builder PR. + +#### Pretesting Manifest-Only connectors + +Once the publish pipeline has completed, choose a connector to test. Set the base_image in the connector's metadata to your pre-release version in Dockerhub (make sure to update the SHA as well). +Next, build the pre-release image locally using `airbyte-ci connectors —name= build`. +You can now run connector interfaces against the built image using the pattern
`docker run airbyte/:dev `. +The connector's README should include a list of these commands, which can be copy/pasted and run from the connector's directory for quick testing against a local config. +You can also run `airbyte-ci connectors —name= test` to run the CI test suite against the dev image. + +#### Pretesting Low-Code Python connectors + +Once the publish pipeline has completed, set the version of `airbyte-cdk` in the connector's pyproject.toml file to the pre-release version in PyPI. +Update the lockfile and run connector interfaces via poetry:
`poetry run source- spec/check/discover/read`. +You can also run `airbyte-ci connectors —name= test` to run the CI test suite against the dev image.

 + +#### Pretesting in Cloud + +It is possible to pretest a version of SDM in Airbyte Cloud using the following steps: + +1. Publish a pre-release version. +2. Open Cloud and create a custom source in the Builder (ie fork PokeAPI with no changes). Publish the source to your workspace. +3. Set up a connection using the forked custom source. +4. Connect to the production database with a tool like DBeaver. +5. Manually update the connector's `actor_definition_version`. + +Because this process requires accessing and updating the production database manually, it is NOT RECOMMENDED for most cases. Only do so if you understand the risks, are already confident navigating the database, and feel the potential risk of your changes breaking the CDK/SDM is high enough to warrant this process.