Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1117,8 +1117,7 @@ repos:
(?x)
^dev/breeze/.*$|
^\.pre-commit-config\.yaml$|
^scripts/ci/prek/breeze_cmd_line\.py$|
^generated/provider_dependencies\.json$
^scripts/ci/prek/breeze_cmd_line\.py$$
require_serial: true
pass_filenames: false
- id: check-example-dags-urls
Expand Down
29 changes: 5 additions & 24 deletions dev/breeze/doc/ci/04_selective_checks.md
Original file line number Diff line number Diff line change
Expand Up @@ -312,8 +312,7 @@ all versions), the cause is almost always a single rule that fired. To find it:
pushing.
3. **Check the usual escalation triggers** (any one of these forces the full matrix):
* an **environment file** changed — `.github/workflows/*`, `dev/breeze/src/*`, `Dockerfile*`,
`scripts/ci/*`, `scripts/docker/*`, `generated/provider_dependencies.json` (often this is the
surprise: editing CI/breeze itself runs everything);
`scripts/ci/*`, `scripts/docker/*`, (often this is the surprise: editing CI/breeze itself runs everything);
* **`pyproject.toml`** or generated provider dependencies changed (also forces `all_versions`);
* the **generated OpenAPI spec** or the client generator changed (the API contract);
* **`tests/utils`** or **git/standard provider** files changed;
Expand Down Expand Up @@ -387,8 +386,8 @@ together using `pytest-xdist` (pytest-xdist distributes the tests among parallel

* `Full tests` case is enabled when the event is PUSH **to `main`**, SCHEDULE or WORKFLOW_DISPATCH, or we
miss commit info, or any of the important environment files (`pyproject.toml`, `Dockerfile`, `scripts`,
`generated/provider_dependencies.json` etc.) changed, or the API *contract* changed (the generated
OpenAPI spec or the client generator — plain API source/test edits that leave the committed spec
etc.) changed, or the API *contract* changed (the generated OpenAPI spec or the client generator —
plain API source/test edits that leave the committed spec
untouched do **not** force full tests), or `tests/utils` / git / standard provider files changed, or
when the `full tests needed` label is set.
That enables all matrix combinations of variables (representative) and all possible test type. No further
Expand Down Expand Up @@ -556,23 +555,6 @@ GitHub Actions to pass the list of parameters to a command to execute
| upgrade-to-newer-dependencies | Whether the image build should attempt to upgrade all dependencies (true/false or commit hash) | false | |


[1] Note for deciding if `full tests needed` mode is enabled and provider.yaml files.

When we decided whether to run `full tests` we do not check (directly) if provider.yaml files changed,
even if they are single source of truth for provider dependencies and when you add a dependency there,
the environment changes and generally full tests are advised.

This is because provider.yaml change will automatically trigger (via `update-provider-dependencies` prek)
generation of `generated/provider_dependencies.json` and `pyproject.toml` gets updated as well. This is a far
better indication if we need to run full tests than just checking if provider.yaml files changed, because
provider.yaml files contain more information than just dependencies - they are the single source of truth
for a lot of information for each provider and sometimes (for example when we update provider documentation
or when new Hook class is added), we do not need to run full tests.

That's why we do not base our `full tests needed` decision on changes in dependency files that are generated
from the `provider.yaml` files, but on `generated/provider_dependencies.json` and `pyproject.toml` files being
modified. This can be overridden by setting `full tests needed` label in the PR.

## Committer vs. Non-committer PRs

There is a difference in how the CI jobs are run for committer and non-committer PRs from forks.
Expand All @@ -595,16 +577,15 @@ builds that also have `canary` label set to also switch the `canary` builds to p
If you are testing CI workflow changes and want to test it for more complete matrix combinations generated by
the jobs - you can set `all versions` label in the PR. This will run the PRs with the same combinations
of versions as the `canary` main build. Using `all versions` is automatically set when build dependencies
change in `pyproject.toml` or when dependencies change for providers in `generated/provider_dependencies.json`
or when `hatch_build.py` changes.
change in `pyproject.toml`.

If you are running an `apache` PR, you can also set `canary` label for such PR and in this case, all the
`canary` properties of build will be used: `self-hosted` runners, `full tests needed` mode, `all versions`
as well as all canary-specific jobs will run there. You can modify this behaviour of the `canary` run by
applying `use public runners`, and `default versions only` labels to the PR as well which will still run
a `canary` equivalent build but with public runners an default Python/K8S versions only - respectively.

If you are testing CI workflow changes and change `pyproject.toml` or `generated/provider_dependencies.json`
If you are testing CI workflow changes and change `pyproject.toml`
and you want to limit the number of matrix combinations generated by
the jobs - you can set `default versions only` label in the PR. This will limit the number of versions
used in the matrix to the default ones (default Python version and default Kubernetes version).
Expand Down
8 changes: 0 additions & 8 deletions dev/breeze/src/airflow_breeze/utils/selective_checks.py
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,6 @@ def __hash__(self):
r"^scripts/ci/prek",
r"^scripts/docker",
r"^scripts/in_container",
r"^generated/provider_dependencies.json$",
],
FileGroupForCi.BREEZE_INTEGRATION_TEST_FILES: [
r"^dev/breeze/src/.*",
Expand Down Expand Up @@ -699,9 +698,6 @@ def _should_run_all_tests_and_versions(self) -> bool:
if self.pyproject_toml_changed:
console_print("[warning]Running everything with all versions: changed pyproject.toml[/]")
return True
if self.generated_dependencies_changed:
console_print("[warning]Running everything with all versions: provider dependencies changed[/]")
return True
return False

@cached_property
Expand Down Expand Up @@ -1404,10 +1400,6 @@ def _print_diff(old_lines: list[str], new_lines: list[str]):
diff = "\n".join(line for line in difflib.ndiff(old_lines, new_lines) if line and line[0] in "+-?")
console_print(diff)

@cached_property
def generated_dependencies_changed(self) -> bool:
return "generated/provider_dependencies.json" in self._files

@cached_property
def any_provider_yaml_or_pyproject_toml_changed(self) -> bool:
if not self._commit_ref:
Expand Down
Loading
Loading