Task Refresh from db before task defer#1
Open
purnachandergit wants to merge 111 commits into
Open
Conversation
Co-authored-by: adesai <adesai@cloudera.com>
…entation (apache#40380) * Adding version while releasing providers using prepare-provider-documentation * fixup! Adding version while releasing providers using prepare-provider-documentation --------- Co-authored-by: adesai <adesai@cloudera.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
This test seems to intermittently fail and it looks like it's because timeout when run in parallel. Increasing timeout should help, also logging the timeout should help to diagnose things in case it happens again.
The codespace button takes you to a create codespace page which is much easier to use than read through the setup page which might be an older doc.
The hatch command hatch env shell probably was meant to be hatch shell. Since env is a separate command, this was fixed in the local_virtualenv doc but missed here.
…he#40332) * Add azure identity to get bearer token * Add azure-identity as an additional-extras dependency * Fix tests --------- Co-authored-by: Marcel Martinelli <marcel.martinelli@rabobank.nl>
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
When parallell tests start on CI, they are running parallel docker compose's and on a clean machine in CI this means that every parallel test is pulling the images needed to run tests. This means that backend images are pulled in parallell by all starting parallel runs. This PR optimizes this step - before running the tests in parallel we run `docker compose pull` once with the same compose files as tests - this will pull the necessary images only once. It should save a few seconds and save a lot of unnecessary traffic for CI tests - where same image is pulled multiple times - especially for self-hosted runners of ours where we run 8 docker compose instances in parallel.
* Resolve postgres deprecations in tests * Resolve postgres deprecations in tests
* fix: typo in example_params_ui_tutorial.py (des -> does) Modify "des not allow" to "does not allow" * fix typo in example of DAG parameters Modify a line of example_params_ui_tutorial.py to remove a trailing '.e' at the end of a comment. Before: # Note: Value display mapping does not need to be complete.s After: # Note: Value display mapping does not need to be complete
apache#40353)" (apache#40402) This reverts commit fbcee8d.
Signed-off-by: Kacper Muda <mudakacper@gmail.com>
Using mysql backend needs to compute a connection string without scheme which using the urlJoin templating function left the // characters at the beginning of the string Signed-off-by: Romain Forlot <romain.forlot@dailymotion.com>
The default config only allows for 1 minute for each component to start up.
* add persistentVolumeClaimRetentionPolicy * add new triggerer persistentVolumeClaimRetentionPolicy to values schema * add persistentVolumeClaimRetentionPolicy * revert pod-template-file * add description to values schema * add description to values.json * fix values-schema, imporove comments --------- Co-authored-by: Nir Rosenthal <nirr@perion.com>
* chore: Add param support for client_reqquest_mfa_token * chore: Update test * chore: Fix test param beat extra * chore: Update get_params for client_request_mfa_token * chore: Update client_request_mfa_token to be added if it is set to be True * chore: Add client_request_mfa_token back to the test * chore: Update the test as boolean for cient_request_mfa_token * chore: Cleanup * chore: Exclude client request mfa token as part of uri * style: Lint
Corrected several typographical and grammatical errors in the INSTALL documentation. These changes improve the clarity and readability of the instructions. Specific changes include correcting "CPyhon" to "CPython" and standardizing the capitalization of "Airflow".
* update min weaviate-client version to 4.4.0
* weaviate hook
* update airflow connection to v4 style
* migrate the following methods to v4 API
* `test_connections`
* `create_class` to `create_collection`
* `get_schema` to `get_collection_configuraiton`
* `delete_classes` to `delete_collections`
* `query_with_vector`
* `query_without_vector` to `query_with_text`
* `create_object`
* `get_object`
* `delete_object`
* `update_object`
* `replace_object`
* `object_exists`
* `batch_data`
* `get_or_create_object`
* `update_collection_configuration`
* `_generate_uuids`
* `create_or_replace_document_objects`
* `_delete_objects`
* `_delete_all_documents_objects`
* `_get_documents_to_uuid_map`
* `_get_segregated_documents`
* remove the following methods which are no longer supported in v4 API
* `validate_object`
* `update_schema`
* `create_schema`
* `delete_all_schema`
* `create_or_replace_classes`
* `_compare_schema_subset`
* `_convert_properties_to_dict`
* `check_subset_of_schema`
* remove deprecated method `get_client`
* remove unused argument retry_status_codes in `__init__`
* weaviate operators
* migrate `WeaviateDocumentIngestOperator` and `WeaviateIngestOperator` to use hook with v4 API
…on in deferrable mode (apache#40457) Signed-off-by: Kacper Muda <mudakacper@gmail.com>
…a expression values (apache#40471)
* Update Breeze docs to handle bad interpreter error * move to troubleshooting
* Resolve azure provider deprecations in tests * Resolve azure provider deprecations in tests
--------- Co-authored-by: Jacob Lee <seungmin_lee3@apple.com>
* Further clarify triage process * update * update
* Be able to remove ACL in S3 hook's copy_object * Add default test as well * Update s3.py
* fix: add project_id to submit_batch_job hook * chore: static checks compliance
- minor rephrase of example
…spend and Resume operations (apache#40509) Support added to Teradata Provider about Teradata Compute Cluster feature Provisioning compute cluster instance Decommissioning compute cluster instance Resume and Suspend of compute cluster instance
…) (apache#40558) * Add support for query parameters to BigQueryCheckOperator (apache#40556) Remove unnecessary space * Add a unit test for BigQueryCheckOperator query params; fix missing 'self' reference * Fix lint (apache#40558) --------- Co-authored-by: Alden S. Page <alden.page@doubleverify.com>
…object (apache#40162)" (apache#40577) This reverts commit 2f2796f.
Co-authored-by: khalidzohaib <hello@zohaibkhalid.com>
Opening this pr to refresh the task instance before deferring the task This helps in getting the latest values from db if some values got updated during execute function. closes: apache#40329
purnachandergit
force-pushed
the
task-refresh-before-task-defer
branch
from
aef4ff7 to
3bc350d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opening this pr to refresh the task instance before deferring the task
This helps in getting the latest values from db if some values got updated during execute function.
closes: apache#40329
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.