[Build] Fix skipping tests when there are only documentation changes and run centos7 & windows cpp build only when there are cpp changes - #10376
Conversation
|
@michaeljmarshall @eolivelli @aahmed-se Please review |
eolivelli
left a comment
There was a problem hiding this comment.
+1
@aahmed-se @merlimat PTAL
52628c1 to
339a1e7
Compare
| # pattern syntax: https://github.com/micromatch/picomatch | ||
| all: | ||
| - '**' | ||
| docs: | ||
| - 'site2/**' | ||
| - 'deployment/**' | ||
| - '.asf.yaml' | ||
| - '*.md' | ||
| - '**/*.md' | ||
|
|
||
| - name: Check changed files | ||
| id: check_changes | ||
| run: echo "::set-output name=docs_only::${{ fromJSON(steps.changes.outputs.all_count) == fromJSON(steps.changes.outputs.docs_count) && fromJSON(steps.changes.outputs.docs_count) > 0 }}" |
There was a problem hiding this comment.
this seems complex, but it's necessary.
picomatch translates a pattern to a regex under the covers. combining including and excluding can lead to odd results.
It's possible to test picomatch patterns using npm on the command line, for example:
# check picomatch version from https://github.com/apache/pulsar-test-infra/blob/master/paths-filter/package-lock.json
npm install picomatch@2.2.1
function test_picomatch_pattern() { node -e 'const pattern=process.argv[1]; const matcher=require("picomatch")(pattern, {"dot": true}); console.log(`pattern: ${pattern}`); process.argv.splice(2).forEach(arg => console.log(`${arg} -> ${matcher(arg)}`));' "$@" }
test_picomatch_pattern '**/*.md' README.md some/path/README.md
This example shows a problem with inversion (!) in the pattern:
❯ test_picomatch_pattern '**/*.md' README.md some/path/README.md
pattern: **/*.md
README.md -> true
some/path/README.md -> true
❯ test_picomatch_pattern '!(**/*.md)' README.md some/path/README.md
pattern: !(**/*.md)
README.md -> true
some/path/README.md -> false
That is the reason why the logic for checking documentation only changes is done based on the comparison on match counts,
fromJSON(steps.changes.outputs.all_count) == fromJSON(steps.changes.outputs.docs_count) && fromJSON(steps.changes.outputs.docs_count) > 0 .
| pull_request: | ||
| branches: | ||
| - master | ||
| paths-ignore: ['site2/**', 'deployment/**'] |
There was a problem hiding this comment.
paths-ignore would be fine here, but it's removed to be consistent with other workflows in apache/pulsar where paths-filter action (copy of https://github.com/dorny/paths-filter) is used.
GitHub provides a paths-ignore feature out of the box.
However, GitHub'spaths-ignorehas some limitations:
- GitHub's
paths-ignorefails to look at previous commits. This means that the outcome depends on how often you push changes.- Consequently, GitHub's
paths-ignoredoes not work for required checks.
If you path-ignore a required check, then pull requests will block forever without being mergeable.
(source)
paths-ignore was once used in apache/pulsar (introduced by #9527), but it had to be reverted in #9606 because of the limitations.
|
/pulsarbot run-failure-checks |
Fixes #10373
Motivation
Tests should be skipped for PR builds that only contain documentation changes. This is currently broken.
A new build has been introduced for building the CPP client on Windows. Make the changes in
ci-cpp-build-centos7.yamlandci-cpp-build-windows.yamlso that the builds run only when there are changes in CPP client files (or the workflow definitions).Modifications
Refactor logic to be less error prone. Use
fromJSONfunction to parse a number in a string as a number.Additional Context
#10276