DATA-629: Added Airbyte configuration documentation#97
Conversation
| - General: ``config/<ENV>/ingestion.yml`` | ||
| - Ingestion tool related: e.g. ``config/<ENV>/airbyte.yml`` | ||
|
|
||
| ``config/<ENV>/bi.yml`` contains basic configuration of ingestion: |
There was a problem hiding this comment.
bi.yml?
| - Flag for enable/disable ingestion option in **dp**. | ||
| * - engine | ||
| - enum string | ||
| - BI tool you want to working with (currently only supported value is ``airbyte``). |
There was a problem hiding this comment.
BI tool?
| - array<*task*> | ||
| - configurations of Airflow tasks used by `dbt-airflow-factory <https://github.com/getindata/dbt-airflow-factory>`_. | ||
| Allowed *task* options are documented `here <https://dbt-airflow-factory.readthedocs.io/en/latest/configuration.html#id3>`_. | ||
|
|
There was a problem hiding this comment.
Missing example - the config structure is a bit complex and needs to be supported with an example.
There was a problem hiding this comment.
ok
|
|
||
| Business Intelligence configuration | ||
| ++++++++++++++++++++++++++++++ | ||
|
|
There was a problem hiding this comment.
BI config should also be supported with an example
There was a problem hiding this comment.
added example
| * - tasks | ||
| - array<*task*> | ||
| - configurations of Airflow tasks used by `dbt-airflow-factory <https://github.com/getindata/dbt-airflow-factory>`_. | ||
| Allowed *task* options are documented `here <https://dbt-airflow-factory.readthedocs.io/en/latest/configuration.html#id3>`_. |
There was a problem hiding this comment.
is this configuration up-to-date? https://dbt-airflow-factory.readthedocs.io/en/latest/configuration.html#id3
There was a problem hiding this comment.
afaik it is. Even with a typo asyncronous that is now widespread
31302a3 to
c042eb4
Compare
|
Michał Rudko (@michalrudko) I updated documentation with some examples as well as divided airbyte.yml into two parts (marked one as not required). Compiled version of configuration.rst file looks like this atm: |
* Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * feat: looker integration for compile and deploy, new command: bi * tests fix, looker git with ssh * chore: organizing imports, removed not used variables * chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path * chore: formatting, types-requests dependency for mypy * refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option * Fix after rebase * Fix after rebase * Fix after rebase * Fix after rebase * Airbyte integration (#86) * [DATA-366] Update datahub package and add dbt extra (#79) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Added IAP Airbyte authorization with service account file Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> * Service account OIDC ID token in deploy command (#91) * Added possibility to pass platform-agnostic service account OIDC ID token to deploy command * Added changelog information * added 'source freshness' command to dbt compile * Bump copier (#93) * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * dp compile default environment has been set to local * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump datahub (#84) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump packages except copier * fix: package name * chore: add specific version for pyyaml * fix: misspelling * chore: test docs * chore: test * test * test * test * test * test * test * chore: bump packages * chore: add renovate * chore: bump copier * chore: update exception * fix: update by removing not existing flag * chore: update tests * feat: update schema on manifest to v7 * chore: bump packages * chore: delete renovate * chore: bump python version * chore: remove junk * chore: bring back old exception * chore: add matrix to tox * chore: add new require and classifier * trigger * chore(docs): update badge and changelog * chore(docs): replace old extension * chore: add description * chore(docs): add old suffix and envops * chore(docs): update changelog * chore(docs): update changelog * chore(docs): update copier * chore: pinup version for dbt2looker * chore(docs): update changelog * chore: replace var for more meaningful Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> * chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * FIX #103 - Bump version and CHANGELOG for release 0.25.0 --------- Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Piotr Sierkin <psierkin@gmail.com> Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com>
* chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Fix failing readthedocs builds (#104) * Removed sphinx action from prepare-release pipeline (#105) * Bump werkzeug from 2.2.2 to 2.2.3 (#106) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@2.2.2...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * bump codeclimate-action to 4.0.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> * FIX #108 - Bump version and CHANGELOG for release 0.25.1 * removed trailing characters from CHANGELOG.md --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Piotr Sierkin <psierkin@gmail.com> Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * feat: looker integration for compile and deploy, new command: bi * tests fix, looker git with ssh * chore: organizing imports, removed not used variables * chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path * chore: formatting, types-requests dependency for mypy * refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option * Fix after rebase * Fix after rebase * Fix after rebase * Fix after rebase * Airbyte integration (#86) * [DATA-366] Update datahub package and add dbt extra (#79) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Added IAP Airbyte authorization with service account file Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> * Service account OIDC ID token in deploy command (#91) * Added possibility to pass platform-agnostic service account OIDC ID token to deploy command * Added changelog information * added 'source freshness' command to dbt compile * Bump copier (#93) * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * dp compile default environment has been set to local * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump datahub (#84) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump packages except copier * fix: package name * chore: add specific version for pyyaml * fix: misspelling * chore: test docs * chore: test * test * test * test * test * test * test * chore: bump packages * chore: add renovate * chore: bump copier * chore: update exception * fix: update by removing not existing flag * chore: update tests * feat: update schema on manifest to v7 * chore: bump packages * chore: delete renovate * chore: bump python version * chore: remove junk * chore: bring back old exception * chore: add matrix to tox * chore: add new require and classifier * trigger * chore(docs): update badge and changelog * chore(docs): replace old extension * chore: add description * chore(docs): add old suffix and envops * chore(docs): update changelog * chore(docs): update changelog * chore(docs): update copier * chore: pinup version for dbt2looker * chore(docs): update changelog * chore: replace var for more meaningful Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> * chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Fix failing readthedocs builds (#104) * Removed sphinx action from prepare-release pipeline (#105) * Bump werkzeug from 2.2.2 to 2.2.3 (#106) * Bump werkzeug from 2.2.2 to 2.2.3 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@2.2.2...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * bump codeclimate-action to 4.0.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> * Airbyte fix creating multiple connections (#109) * Fixing tests. * Fix failing test. * Remove redundant line. * Add info about fix to CHANGELOG.md. * Add workspace_id entry to config. * Get rid of walrus operator & add test for missing workspace id. * Fix wrong connection id in Airflow ingestion task. * Fix typo in test name. * Add fetching default workspace if not available in config. * Increasign test coverage. * Update workspace_id configuration description. * Bumping Python version to 3.8 in prepare-release workflow. * Pin pydantic version (#112) * Pin Pydantic version until Copier fixes their dependency. * Add info to changelog. * FIX #113 - Bump version and CHANGELOG for release 0.25.3 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Piotr Sierkin <psierkin@gmail.com> Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jakub Szafran <33091690+jszafran@users.noreply.github.com> Co-authored-by: jszafran <jszafran.pv@gmail.com>
* Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * feat: looker integration for compile and deploy, new command: bi * tests fix, looker git with ssh * chore: organizing imports, removed not used variables * chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path * chore: formatting, types-requests dependency for mypy * refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option * Fix after rebase * Fix after rebase * Fix after rebase * Fix after rebase * Airbyte integration (#86) * [DATA-366] Update datahub package and add dbt extra (#79) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Added IAP Airbyte authorization with service account file Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> * Service account OIDC ID token in deploy command (#91) * Added possibility to pass platform-agnostic service account OIDC ID token to deploy command * Added changelog information * added 'source freshness' command to dbt compile * Bump copier (#93) * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * dp compile default environment has been set to local * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump datahub (#84) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump packages except copier * fix: package name * chore: add specific version for pyyaml * fix: misspelling * chore: test docs * chore: test * test * test * test * test * test * test * chore: bump packages * chore: add renovate * chore: bump copier * chore: update exception * fix: update by removing not existing flag * chore: update tests * feat: update schema on manifest to v7 * chore: bump packages * chore: delete renovate * chore: bump python version * chore: remove junk * chore: bring back old exception * chore: add matrix to tox * chore: add new require and classifier * trigger * chore(docs): update badge and changelog * chore(docs): replace old extension * chore: add description * chore(docs): add old suffix and envops * chore(docs): update changelog * chore(docs): update changelog * chore(docs): update copier * chore: pinup version for dbt2looker * chore(docs): update changelog * chore: replace var for more meaningful Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> * chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Fix failing readthedocs builds (#104) * Removed sphinx action from prepare-release pipeline (#105) * Bump werkzeug from 2.2.2 to 2.2.3 (#106) * Bump werkzeug from 2.2.2 to 2.2.3 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@2.2.2...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * bump codeclimate-action to 4.0.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> * Airbyte fix creating multiple connections (#109) * Fixing tests. * Fix failing test. * Remove redundant line. * Add info about fix to CHANGELOG.md. * Add workspace_id entry to config. * Get rid of walrus operator & add test for missing workspace id. * Fix wrong connection id in Airflow ingestion task. * Fix typo in test name. * Add fetching default workspace if not available in config. * Increasign test coverage. * Update workspace_id configuration description. * Bumping Python version to 3.8 in prepare-release workflow. * Pin pydantic version (#112) * Pin Pydantic version until Copier fixes their dependency. * Add info to changelog. * Upgrade dbt to 1.5.4 (#114) * Upgrade dbt to 1.5.4 * Fixing imports for publish.py * Update of dbt_schema_version * Updating manifest.json for tests * Fix manifest for testing * Fixing manifest cntd. * Fixing manifest cntd. * Replace example-project with exampleproject * Adding tutorial for upgrading manifest in tests cntd. * Fixing typos in the docs * Version fixes before release --------- Co-authored-by: Michal Rudko <michal.rudko@getindata.com> * FIX #115 - Bump version and CHANGELOG for release 0.26.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Piotr Sierkin <psierkin@gmail.com> Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jakub Szafran <33091690+jszafran@users.noreply.github.com> Co-authored-by: jszafran <jszafran.pv@gmail.com> Co-authored-by: michalrudko <michal.rudko@gmail.com> Co-authored-by: Michal Rudko <michal.rudko@getindata.com>
* Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * feat: looker integration for compile and deploy, new command: bi * tests fix, looker git with ssh * chore: organizing imports, removed not used variables * chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path * chore: formatting, types-requests dependency for mypy * refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option * Fix after rebase * Fix after rebase * Fix after rebase * Fix after rebase * Airbyte integration (#86) * [DATA-366] Update datahub package and add dbt extra (#79) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Added IAP Airbyte authorization with service account file Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> * Service account OIDC ID token in deploy command (#91) * Added possibility to pass platform-agnostic service account OIDC ID token to deploy command * Added changelog information * added 'source freshness' command to dbt compile * Bump copier (#93) * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * dp compile default environment has been set to local * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump datahub (#84) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump packages except copier * fix: package name * chore: add specific version for pyyaml * fix: misspelling * chore: test docs * chore: test * test * test * test * test * test * test * chore: bump packages * chore: add renovate * chore: bump copier * chore: update exception * fix: update by removing not existing flag * chore: update tests * feat: update schema on manifest to v7 * chore: bump packages * chore: delete renovate * chore: bump python version * chore: remove junk * chore: bring back old exception * chore: add matrix to tox * chore: add new require and classifier * trigger * chore(docs): update badge and changelog * chore(docs): replace old extension * chore: add description * chore(docs): add old suffix and envops * chore(docs): update changelog * chore(docs): update changelog * chore(docs): update copier * chore: pinup version for dbt2looker * chore(docs): update changelog * chore: replace var for more meaningful Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> * chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Fix failing readthedocs builds (#104) * Removed sphinx action from prepare-release pipeline (#105) * Bump werkzeug from 2.2.2 to 2.2.3 (#106) * Bump werkzeug from 2.2.2 to 2.2.3 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@2.2.2...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * bump codeclimate-action to 4.0.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> * Airbyte fix creating multiple connections (#109) * Fixing tests. * Fix failing test. * Remove redundant line. * Add info about fix to CHANGELOG.md. * Add workspace_id entry to config. * Get rid of walrus operator & add test for missing workspace id. * Fix wrong connection id in Airflow ingestion task. * Fix typo in test name. * Add fetching default workspace if not available in config. * Increasign test coverage. * Update workspace_id configuration description. * Bumping Python version to 3.8 in prepare-release workflow. * Pin pydantic version (#112) * Pin Pydantic version until Copier fixes their dependency. * Add info to changelog. * Upgrade dbt to 1.5.4 (#114) * Upgrade dbt to 1.5.4 * Fixing imports for publish.py * Update of dbt_schema_version * Updating manifest.json for tests * Fix manifest for testing * Fixing manifest cntd. * Fixing manifest cntd. * Replace example-project with exampleproject * Adding tutorial for upgrading manifest in tests cntd. * Fixing typos in the docs * Version fixes before release --------- Co-authored-by: Michal Rudko <michal.rudko@getindata.com> * Databricks generate job (#116) * new feature * small fixes * dbt databricks factory requirement * python 3.9 onwards * testing * linting * annotations fix * linting * no docs * linting * update docs * testing * tests with databricks * testenv * Upgrade pyyaml * Dbt glue (#118) * DBT glue * DBT glue * DBT glue * DBT glue * FIX #119 - Bump version and CHANGELOG for release 0.27.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Piotr Sierkin <psierkin@gmail.com> Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jakub Szafran <33091690+jszafran@users.noreply.github.com> Co-authored-by: jszafran <jszafran.pv@gmail.com> Co-authored-by: michalrudko <michal.rudko@gmail.com> Co-authored-by: Michal Rudko <michal.rudko@getindata.com> Co-authored-by: ptutak-getindata <133227844+ptutak-getindata@users.noreply.github.com>
* Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * feat: looker integration for compile and deploy, new command: bi * tests fix, looker git with ssh * chore: organizing imports, removed not used variables * chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path * chore: formatting, types-requests dependency for mypy * refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option * Fix after rebase * Fix after rebase * Fix after rebase * Fix after rebase * Airbyte integration (#86) * [DATA-366] Update datahub package and add dbt extra (#79) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Added IAP Airbyte authorization with service account file Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> * Service account OIDC ID token in deploy command (#91) * Added possibility to pass platform-agnostic service account OIDC ID token to deploy command * Added changelog information * added 'source freshness' command to dbt compile * Bump copier (#93) * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * dp compile default environment has been set to local * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump datahub (#84) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump packages except copier * fix: package name * chore: add specific version for pyyaml * fix: misspelling * chore: test docs * chore: test * test * test * test * test * test * test * chore: bump packages * chore: add renovate * chore: bump copier * chore: update exception * fix: update by removing not existing flag * chore: update tests * feat: update schema on manifest to v7 * chore: bump packages * chore: delete renovate * chore: bump python version * chore: remove junk * chore: bring back old exception * chore: add matrix to tox * chore: add new require and classifier * trigger * chore(docs): update badge and changelog * chore(docs): replace old extension * chore: add description * chore(docs): add old suffix and envops * chore(docs): update changelog * chore(docs): update changelog * chore(docs): update copier * chore: pinup version for dbt2looker * chore(docs): update changelog * chore: replace var for more meaningful Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> * chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Fix failing readthedocs builds (#104) * Removed sphinx action from prepare-release pipeline (#105) * Bump werkzeug from 2.2.2 to 2.2.3 (#106) * Bump werkzeug from 2.2.2 to 2.2.3 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@2.2.2...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * bump codeclimate-action to 4.0.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> * Airbyte fix creating multiple connections (#109) * Fixing tests. * Fix failing test. * Remove redundant line. * Add info about fix to CHANGELOG.md. * Add workspace_id entry to config. * Get rid of walrus operator & add test for missing workspace id. * Fix wrong connection id in Airflow ingestion task. * Fix typo in test name. * Add fetching default workspace if not available in config. * Increasign test coverage. * Update workspace_id configuration description. * Bumping Python version to 3.8 in prepare-release workflow. * Pin pydantic version (#112) * Pin Pydantic version until Copier fixes their dependency. * Add info to changelog. * Upgrade dbt to 1.5.4 (#114) * Upgrade dbt to 1.5.4 * Fixing imports for publish.py * Update of dbt_schema_version * Updating manifest.json for tests * Fix manifest for testing * Fixing manifest cntd. * Fixing manifest cntd. * Replace example-project with exampleproject * Adding tutorial for upgrading manifest in tests cntd. * Fixing typos in the docs * Version fixes before release --------- Co-authored-by: Michal Rudko <michal.rudko@getindata.com> * Databricks generate job (#116) * new feature * small fixes * dbt databricks factory requirement * python 3.9 onwards * testing * linting * annotations fix * linting * no docs * linting * update docs * testing * tests with databricks * testenv * Upgrade pyyaml * Dbt glue (#118) * DBT glue * DBT glue * DBT glue * DBT glue * Dbt glue with spark (#120) * Dbt glue with spark * Fixing dp cli * FIX #121 - Bump version and CHANGELOG for release 0.28.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Piotr Sierkin <psierkin@gmail.com> Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jakub Szafran <33091690+jszafran@users.noreply.github.com> Co-authored-by: jszafran <jszafran.pv@gmail.com> Co-authored-by: michalrudko <michal.rudko@gmail.com> Co-authored-by: Michal Rudko <michal.rudko@getindata.com> Co-authored-by: ptutak-getindata <133227844+ptutak-getindata@users.noreply.github.com>
* Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * feat: looker integration for compile and deploy, new command: bi * tests fix, looker git with ssh * chore: organizing imports, removed not used variables * chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path * chore: formatting, types-requests dependency for mypy * refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option * Fix after rebase * Fix after rebase * Fix after rebase * Fix after rebase * Airbyte integration (#86) * [DATA-366] Update datahub package and add dbt extra (#79) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Added IAP Airbyte authorization with service account file Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> * Service account OIDC ID token in deploy command (#91) * Added possibility to pass platform-agnostic service account OIDC ID token to deploy command * Added changelog information * added 'source freshness' command to dbt compile * Bump copier (#93) * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * dp compile default environment has been set to local * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump datahub (#84) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump packages except copier * fix: package name * chore: add specific version for pyyaml * fix: misspelling * chore: test docs * chore: test * test * test * test * test * test * test * chore: bump packages * chore: add renovate * chore: bump copier * chore: update exception * fix: update by removing not existing flag * chore: update tests * feat: update schema on manifest to v7 * chore: bump packages * chore: delete renovate * chore: bump python version * chore: remove junk * chore: bring back old exception * chore: add matrix to tox * chore: add new require and classifier * trigger * chore(docs): update badge and changelog * chore(docs): replace old extension * chore: add description * chore(docs): add old suffix and envops * chore(docs): update changelog * chore(docs): update changelog * chore(docs): update copier * chore: pinup version for dbt2looker * chore(docs): update changelog * chore: replace var for more meaningful Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> * chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Fix failing readthedocs builds (#104) * Removed sphinx action from prepare-release pipeline (#105) * Bump werkzeug from 2.2.2 to 2.2.3 (#106) * Bump werkzeug from 2.2.2 to 2.2.3 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@2.2.2...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * bump codeclimate-action to 4.0.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> * Airbyte fix creating multiple connections (#109) * Fixing tests. * Fix failing test. * Remove redundant line. * Add info about fix to CHANGELOG.md. * Add workspace_id entry to config. * Get rid of walrus operator & add test for missing workspace id. * Fix wrong connection id in Airflow ingestion task. * Fix typo in test name. * Add fetching default workspace if not available in config. * Increasign test coverage. * Update workspace_id configuration description. * Bumping Python version to 3.8 in prepare-release workflow. * Pin pydantic version (#112) * Pin Pydantic version until Copier fixes their dependency. * Add info to changelog. * Upgrade dbt to 1.5.4 (#114) * Upgrade dbt to 1.5.4 * Fixing imports for publish.py * Update of dbt_schema_version * Updating manifest.json for tests * Fix manifest for testing * Fixing manifest cntd. * Fixing manifest cntd. * Replace example-project with exampleproject * Adding tutorial for upgrading manifest in tests cntd. * Fixing typos in the docs * Version fixes before release --------- Co-authored-by: Michal Rudko <michal.rudko@getindata.com> * Databricks generate job (#116) * new feature * small fixes * dbt databricks factory requirement * python 3.9 onwards * testing * linting * annotations fix * linting * no docs * linting * update docs * testing * tests with databricks * testenv * Upgrade pyyaml * Dbt glue (#118) * DBT glue * DBT glue * DBT glue * DBT glue * Dbt glue with spark (#120) * Dbt glue with spark * Fixing dp cli * Dbt upgtade (#122) * Dbt upgrade to 1.7 * Dbt upgrade to 1.7 * Fixing dp cli * FIX #123 - Bump version and CHANGELOG for release 0.29.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Piotr Sierkin <psierkin@gmail.com> Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jakub Szafran <33091690+jszafran@users.noreply.github.com> Co-authored-by: jszafran <jszafran.pv@gmail.com> Co-authored-by: michalrudko <michal.rudko@gmail.com> Co-authored-by: Michal Rudko <michal.rudko@getindata.com> Co-authored-by: ptutak-getindata <133227844+ptutak-getindata@users.noreply.github.com>
* Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * feat: looker integration for compile and deploy, new command: bi * tests fix, looker git with ssh * chore: organizing imports, removed not used variables * chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path * chore: formatting, types-requests dependency for mypy * refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option * Fix after rebase * Fix after rebase * Fix after rebase * Fix after rebase * Airbyte integration (#86) * [DATA-366] Update datahub package and add dbt extra (#79) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Added IAP Airbyte authorization with service account file Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> * Service account OIDC ID token in deploy command (#91) * Added possibility to pass platform-agnostic service account OIDC ID token to deploy command * Added changelog information * added 'source freshness' command to dbt compile * Bump copier (#93) * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * dp compile default environment has been set to local * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump datahub (#84) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump packages except copier * fix: package name * chore: add specific version for pyyaml * fix: misspelling * chore: test docs * chore: test * test * test * test * test * test * test * chore: bump packages * chore: add renovate * chore: bump copier * chore: update exception * fix: update by removing not existing flag * chore: update tests * feat: update schema on manifest to v7 * chore: bump packages * chore: delete renovate * chore: bump python version * chore: remove junk * chore: bring back old exception * chore: add matrix to tox * chore: add new require and classifier * trigger * chore(docs): update badge and changelog * chore(docs): replace old extension * chore: add description * chore(docs): add old suffix and envops * chore(docs): update changelog * chore(docs): update changelog * chore(docs): update copier * chore: pinup version for dbt2looker * chore(docs): update changelog * chore: replace var for more meaningful Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> * chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Fix failing readthedocs builds (#104) * Removed sphinx action from prepare-release pipeline (#105) * Bump werkzeug from 2.2.2 to 2.2.3 (#106) * Bump werkzeug from 2.2.2 to 2.2.3 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@2.2.2...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * bump codeclimate-action to 4.0.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> * Airbyte fix creating multiple connections (#109) * Fixing tests. * Fix failing test. * Remove redundant line. * Add info about fix to CHANGELOG.md. * Add workspace_id entry to config. * Get rid of walrus operator & add test for missing workspace id. * Fix wrong connection id in Airflow ingestion task. * Fix typo in test name. * Add fetching default workspace if not available in config. * Increasign test coverage. * Update workspace_id configuration description. * Bumping Python version to 3.8 in prepare-release workflow. * Pin pydantic version (#112) * Pin Pydantic version until Copier fixes their dependency. * Add info to changelog. * Upgrade dbt to 1.5.4 (#114) * Upgrade dbt to 1.5.4 * Fixing imports for publish.py * Update of dbt_schema_version * Updating manifest.json for tests * Fix manifest for testing * Fixing manifest cntd. * Fixing manifest cntd. * Replace example-project with exampleproject * Adding tutorial for upgrading manifest in tests cntd. * Fixing typos in the docs * Version fixes before release --------- Co-authored-by: Michal Rudko <michal.rudko@getindata.com> * Databricks generate job (#116) * new feature * small fixes * dbt databricks factory requirement * python 3.9 onwards * testing * linting * annotations fix * linting * no docs * linting * update docs * testing * tests with databricks * testenv * Upgrade pyyaml * Dbt glue (#118) * DBT glue * DBT glue * DBT glue * DBT glue * Dbt glue with spark (#120) * Dbt glue with spark * Fixing dp cli * Dbt upgtade (#122) * Dbt upgrade to 1.7 * Dbt upgrade to 1.7 * Fixing dp cli * Upgrade versions * Upgrade versions * FIX #124 - Bump version and CHANGELOG for release 0.30.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Piotr Sierkin <psierkin@gmail.com> Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jakub Szafran <33091690+jszafran@users.noreply.github.com> Co-authored-by: jszafran <jszafran.pv@gmail.com> Co-authored-by: michalrudko <michal.rudko@gmail.com> Co-authored-by: Michal Rudko <michal.rudko@getindata.com> Co-authored-by: ptutak-getindata <133227844+ptutak-getindata@users.noreply.github.com>
* Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * feat: looker integration for compile and deploy, new command: bi * tests fix, looker git with ssh * chore: organizing imports, removed not used variables * chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path * chore: formatting, types-requests dependency for mypy * refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option * Fix after rebase * Fix after rebase * Fix after rebase * Fix after rebase * Airbyte integration (#86) * [DATA-366] Update datahub package and add dbt extra (#79) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Added IAP Airbyte authorization with service account file Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> * Service account OIDC ID token in deploy command (#91) * Added possibility to pass platform-agnostic service account OIDC ID token to deploy command * Added changelog information * added 'source freshness' command to dbt compile * Bump copier (#93) * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * dp compile default environment has been set to local * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump datahub (#84) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump packages except copier * fix: package name * chore: add specific version for pyyaml * fix: misspelling * chore: test docs * chore: test * test * test * test * test * test * test * chore: bump packages * chore: add renovate * chore: bump copier * chore: update exception * fix: update by removing not existing flag * chore: update tests * feat: update schema on manifest to v7 * chore: bump packages * chore: delete renovate * chore: bump python version * chore: remove junk * chore: bring back old exception * chore: add matrix to tox * chore: add new require and classifier * trigger * chore(docs): update badge and changelog * chore(docs): replace old extension * chore: add description * chore(docs): add old suffix and envops * chore(docs): update changelog * chore(docs): update changelog * chore(docs): update copier * chore: pinup version for dbt2looker * chore(docs): update changelog * chore: replace var for more meaningful Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> * chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Fix failing readthedocs builds (#104) * Removed sphinx action from prepare-release pipeline (#105) * Bump werkzeug from 2.2.2 to 2.2.3 (#106) * Bump werkzeug from 2.2.2 to 2.2.3 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@2.2.2...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * bump codeclimate-action to 4.0.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> * Airbyte fix creating multiple connections (#109) * Fixing tests. * Fix failing test. * Remove redundant line. * Add info about fix to CHANGELOG.md. * Add workspace_id entry to config. * Get rid of walrus operator & add test for missing workspace id. * Fix wrong connection id in Airflow ingestion task. * Fix typo in test name. * Add fetching default workspace if not available in config. * Increasign test coverage. * Update workspace_id configuration description. * Bumping Python version to 3.8 in prepare-release workflow. * Pin pydantic version (#112) * Pin Pydantic version until Copier fixes their dependency. * Add info to changelog. * Upgrade dbt to 1.5.4 (#114) * Upgrade dbt to 1.5.4 * Fixing imports for publish.py * Update of dbt_schema_version * Updating manifest.json for tests * Fix manifest for testing * Fixing manifest cntd. * Fixing manifest cntd. * Replace example-project with exampleproject * Adding tutorial for upgrading manifest in tests cntd. * Fixing typos in the docs * Version fixes before release --------- Co-authored-by: Michal Rudko <michal.rudko@getindata.com> * Databricks generate job (#116) * new feature * small fixes * dbt databricks factory requirement * python 3.9 onwards * testing * linting * annotations fix * linting * no docs * linting * update docs * testing * tests with databricks * testenv * Upgrade pyyaml * Dbt glue (#118) * DBT glue * DBT glue * DBT glue * DBT glue * Dbt glue with spark (#120) * Dbt glue with spark * Fixing dp cli * Dbt upgtade (#122) * Dbt upgrade to 1.7 * Dbt upgrade to 1.7 * Fixing dp cli * Upgrade versions * Upgrade versions * feat: Add support for python 3.11 and 3.12 (#125) # Add Python 3.11 and 3.12 Support ## Summary Adds full support for Python 3.11 and 3.12 while maintaining Python 3.9/3.10 compatibility. Python 3.9 is deprecated (EOL Oct 2025) with support ending April 30, 2026. ## Changes **Dependencies** - Removed `pydantic<2` (unused, blocked Python 3.12) - Updated `packaging` to `>=23.0` (setuptools compatibility) - Updated `fsspec`, `gcsfs`, `s3fs` to `>=2024.6.0,<2025.0.0` - Added `setuptools<75.0.0` constraint in tox/requirements-dev **Testing** - Configured tox for `py39`, `py310`, `py311`, `py312` - All 112 tests pass on Python 3.10-3.12 (96% coverage) - Updated pre-commit: flake8 6.1.0, mypy 1.7.1 **Code Quality** - Fixed exception handling: `ClientError` instead of bare `Exception` - Removed unnecessary `type: ignore` comments - Fixed line length violations **Documentation** - Updated `CONTRIBUTING.md` with dev setup and tox usage - Updated `.pre-commit-config.yaml` tool versions ## Testing ```bash tox # All versions tox -e py312 # Specific version Results: 112 tests passed, 96% coverage on Python 3.9-3.12 Release Version: 0.31.0 Type: Minor (non-breaking, adds Python 3.11/3.12 support) Target Branch: develop → main Breaking Changes None. Fully backward compatible. * fix: rollback manual version bump and changelog edit * fix: rollback another manual change in dp version * ci: rollback missing default CHANGELOG configuration that was manually edited * ci: fix release pipeline, add permissions to GH token * ci: fix release pipeline, add permissions to GH token * FIX #126 - Bump version and CHANGELOG for release 0.31.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Piotr Sierkin <psierkin@gmail.com> Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jakub Szafran <33091690+jszafran@users.noreply.github.com> Co-authored-by: jszafran <jszafran.pv@gmail.com> Co-authored-by: michalrudko <michal.rudko@gmail.com> Co-authored-by: Michal Rudko <michal.rudko@getindata.com> Co-authored-by: ptutak-getindata <133227844+ptutak-getindata@users.noreply.github.com> Co-authored-by: Dominik Gniewek-Węgrzyn <47598580+dgniewek@users.noreply.github.com> Co-authored-by: Dominik Gniewek-Węgrzyn <dominik.gniewek@getindata.com>
* Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * feat: looker integration for compile and deploy, new command: bi * tests fix, looker git with ssh * chore: organizing imports, removed not used variables * chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path * chore: formatting, types-requests dependency for mypy * refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option * Fix after rebase * Fix after rebase * Fix after rebase * Fix after rebase * Airbyte integration (#86) * [DATA-366] Update datahub package and add dbt extra (#79) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Added IAP Airbyte authorization with service account file Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> * Service account OIDC ID token in deploy command (#91) * Added possibility to pass platform-agnostic service account OIDC ID token to deploy command * Added changelog information * added 'source freshness' command to dbt compile * Bump copier (#93) * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * dp compile default environment has been set to local * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump datahub (#84) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump packages except copier * fix: package name * chore: add specific version for pyyaml * fix: misspelling * chore: test docs * chore: test * test * test * test * test * test * test * chore: bump packages * chore: add renovate * chore: bump copier * chore: update exception * fix: update by removing not existing flag * chore: update tests * feat: update schema on manifest to v7 * chore: bump packages * chore: delete renovate * chore: bump python version * chore: remove junk * chore: bring back old exception * chore: add matrix to tox * chore: add new require and classifier * trigger * chore(docs): update badge and changelog * chore(docs): replace old extension * chore: add description * chore(docs): add old suffix and envops * chore(docs): update changelog * chore(docs): update changelog * chore(docs): update copier * chore: pinup version for dbt2looker * chore(docs): update changelog * chore: replace var for more meaningful Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> * chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Fix failing readthedocs builds (#104) * Removed sphinx action from prepare-release pipeline (#105) * Bump werkzeug from 2.2.2 to 2.2.3 (#106) * Bump werkzeug from 2.2.2 to 2.2.3 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@2.2.2...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * bump codeclimate-action to 4.0.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> * Airbyte fix creating multiple connections (#109) * Fixing tests. * Fix failing test. * Remove redundant line. * Add info about fix to CHANGELOG.md. * Add workspace_id entry to config. * Get rid of walrus operator & add test for missing workspace id. * Fix wrong connection id in Airflow ingestion task. * Fix typo in test name. * Add fetching default workspace if not available in config. * Increasign test coverage. * Update workspace_id configuration description. * Bumping Python version to 3.8 in prepare-release workflow. * Pin pydantic version (#112) * Pin Pydantic version until Copier fixes their dependency. * Add info to changelog. * Upgrade dbt to 1.5.4 (#114) * Upgrade dbt to 1.5.4 * Fixing imports for publish.py * Update of dbt_schema_version * Updating manifest.json for tests * Fix manifest for testing * Fixing manifest cntd. * Fixing manifest cntd. * Replace example-project with exampleproject * Adding tutorial for upgrading manifest in tests cntd. * Fixing typos in the docs * Version fixes before release --------- Co-authored-by: Michal Rudko <michal.rudko@getindata.com> * Databricks generate job (#116) * new feature * small fixes * dbt databricks factory requirement * python 3.9 onwards * testing * linting * annotations fix * linting * no docs * linting * update docs * testing * tests with databricks * testenv * Upgrade pyyaml * Dbt glue (#118) * DBT glue * DBT glue * DBT glue * DBT glue * Dbt glue with spark (#120) * Dbt glue with spark * Fixing dp cli * Dbt upgtade (#122) * Dbt upgrade to 1.7 * Dbt upgrade to 1.7 * Fixing dp cli * Upgrade versions * Upgrade versions * feat: Add support for python 3.11 and 3.12 (#125) # Add Python 3.11 and 3.12 Support ## Summary Adds full support for Python 3.11 and 3.12 while maintaining Python 3.9/3.10 compatibility. Python 3.9 is deprecated (EOL Oct 2025) with support ending April 30, 2026. ## Changes **Dependencies** - Removed `pydantic<2` (unused, blocked Python 3.12) - Updated `packaging` to `>=23.0` (setuptools compatibility) - Updated `fsspec`, `gcsfs`, `s3fs` to `>=2024.6.0,<2025.0.0` - Added `setuptools<75.0.0` constraint in tox/requirements-dev **Testing** - Configured tox for `py39`, `py310`, `py311`, `py312` - All 112 tests pass on Python 3.10-3.12 (96% coverage) - Updated pre-commit: flake8 6.1.0, mypy 1.7.1 **Code Quality** - Fixed exception handling: `ClientError` instead of bare `Exception` - Removed unnecessary `type: ignore` comments - Fixed line length violations **Documentation** - Updated `CONTRIBUTING.md` with dev setup and tox usage - Updated `.pre-commit-config.yaml` tool versions ## Testing ```bash tox # All versions tox -e py312 # Specific version Results: 112 tests passed, 96% coverage on Python 3.9-3.12 Release Version: 0.31.0 Type: Minor (non-breaking, adds Python 3.11/3.12 support) Target Branch: develop → main Breaking Changes None. Fully backward compatible. * fix: rollback manual version bump and changelog edit * fix: rollback another manual change in dp version * ci: rollback missing default CHANGELOG configuration that was manually edited * ci: fix release pipeline, add permissions to GH token * ci: fix release pipeline, add permissions to GH token * ci: fix publish pipeline - pin a proper version * feat: Expand dbt version support (1.7.x - 1.10.x) and fix dbt 1.8+ compatibility (#127) * feat: support other dbt versions * chore: fix linter issue * FIX #128 - Bump version and CHANGELOG for release 0.32.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Piotr Sierkin <psierkin@gmail.com> Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jakub Szafran <33091690+jszafran@users.noreply.github.com> Co-authored-by: jszafran <jszafran.pv@gmail.com> Co-authored-by: michalrudko <michal.rudko@gmail.com> Co-authored-by: Michal Rudko <michal.rudko@getindata.com> Co-authored-by: ptutak-getindata <133227844+ptutak-getindata@users.noreply.github.com> Co-authored-by: Dominik Gniewek-Węgrzyn <47598580+dgniewek@users.noreply.github.com> Co-authored-by: Dominik Gniewek-Węgrzyn <dominik.gniewek@getindata.com>

<description>Resolves
<issue nr here>Keep in mind: