Skip to content

[Tests] Fix cpp build flakiness by enabling gtest retries, also improve logging in CI #10309

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aahmed-se merged 6 commits into from
Apr 21, 2021
Merged

[Tests] Fix cpp build flakiness by enabling gtest retries, also improve logging in CI #10309

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1e3f1bd
Reduce the number of parallel workers for running cpp tests
lhotari Apr 21, 2021
0ac8929
Store logs for gtests
lhotari Apr 21, 2021
c61d9c5
Copy logs from the docker container to test-logs directory
lhotari Apr 21, 2021
3e706a5
disable color output by default
lhotari Apr 21, 2021
10fb440
Upload test-logs directory if job fails
lhotari Apr 21, 2021
adfe865
Remove "-t" parameter
lhotari Apr 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/ci-cpp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,3 +102,11 @@ jobs:
- name: run c++ tests
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
run: pulsar-client-cpp/docker-tests.sh

- name: Upload test-logs
uses: actions/upload-artifact@v2
if: failure()
continue-on-error: true
with:
name: test-logs
path: test-logs
28 changes: 26 additions & 2 deletions pulsar-client-cpp/docker-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,9 @@ echo "---- Testing Pulsar C++ client using image $IMAGE (type --help for more op

docker pull $IMAGE

DOCKER_CMD="docker run -i -v $ROOT_DIR:/pulsar $IMAGE"
CONTAINER_LABEL="pulsartests=$$"
export GTEST_COLOR=${GTEST_COLOR:-no}
DOCKER_CMD="docker run -e GTEST_COLOR -i -l $CONTAINER_LABEL -v $ROOT_DIR:/pulsar $IMAGE"


for args in "$@"
Expand All @@ -58,4 +60,26 @@ done

# Start 2 Pulsar standalone instances (one with TLS and one without)
# and execute the tests
$DOCKER_CMD bash -c "cd /pulsar/pulsar-client-cpp && ./run-unit-tests.sh ${tests}"
set +e
DISABLE_COLOR_OUTPUT=""
if [ "$GTEST_COLOR" = "no" ]; then
DISABLE_COLOR_OUTPUT="| cat"
fi
$DOCKER_CMD bash -c "cd /pulsar/pulsar-client-cpp && ./run-unit-tests.sh ${tests} $DISABLE_COLOR_OUTPUT"
Comment thread
BewareMyPower marked this conversation as resolved.
RES=$?
if [ $RES -ne 0 ]; then
Comment thread
BewareMyPower marked this conversation as resolved.
(
cd "$ROOT_DIR"
mkdir -p test-logs
cd test-logs
container_id=$(docker ps -a -q --filter "label=$CONTAINER_LABEL")
if [ -n "$container_id" ]; then
# copy logs from the container that ran the tests
docker commit $container_id pulsartests/$container_id
docker run -i --rm pulsartests/$container_id \
bash -c "cd /tmp; tar zcf - gtest-parallel-logs gtest_parallel_results.json pulsar-test-dist/logs" \
| tar zxvf -
fi
)
fi
exit $RES
15 changes: 13 additions & 2 deletions pulsar-client-cpp/run-unit-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,25 @@ cd $ROOT_DIR/pulsar-client-cpp

pushd tests

export RETRY_FAILED="${RETRY_FAILED:-1}"

if [ -f /gtest-parallel/gtest-parallel ]; then
echo "---- Run unit tests in parallel"
gtest_workers=10
# use nproc to set workers to 2 x the number of available cores if nproc is available
if [ -x "$(command -v nproc)" ]; then
gtest_workers=$(( $(nproc) * 2 ))
fi
# set maximum workers to 10
gtest_workers=$(( gtest_workers > 10 ? 10 : gtest_workers ))
echo "---- Run unit tests in parallel (workers=$gtest_workers) (retry_failed=${RETRY_FAILED})"
tests=""
if [ $# -eq 1 ]; then
tests="--gtest_filter=$1"
echo "Running tests: $1"
fi
/gtest-parallel/gtest-parallel ./main $tests --workers=10
/gtest-parallel/gtest-parallel $tests --dump_json_test_results=/tmp/gtest_parallel_results.json \
--workers=$gtest_workers --retry_failed=$RETRY_FAILED -d /tmp \
./main
RES=$?
else
./main
Expand Down