Skip to content

[No QA] Make schema validations async #9814

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aldo-expensify merged 2 commits into from
Jul 12, 2022
+56 −13
Merged

[No QA] Make schema validations async #9814

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
be8a030
Make schema validations async
roryabraham Jul 9, 2022
598fb59
Fix exit code and use shellUtils
roryabraham Jul 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 56 additions & 13 deletions .github/scripts/validateActionsAndWorkflows.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,28 +1,63 @@
#!/bin/bash

echo 'Validates the Github Actions and workflows using the json schemas provided by (https://www.schemastore.org/json/)'
source ./scripts/shellUtils.sh

# Track exit codes separately so we can run a full validation, report errors, and exit with the correct code
declare EXIT_CODE=0
title 'Validating the Github Actions and workflows using the json schemas provided by (https://www.schemastore.org/json/)'

function downloadSchema {
[[ $1 = 'github-action.json' ]] && SCHEMA_NAME='GitHub Action' || SCHEMA_NAME='GitHub Workflow'
info "Downloading $SCHEMA_NAME schema..."
if curl "https://json.schemastore.org/$1" --output "./tempSchemas/$1" --silent; then
success "Successfully downloaded $SCHEMA_NAME schema!"
else
error "Failed downloading $SCHEMA_NAME schema"
exit 1
fi
}

# Download the up-to-date json schemas for github actions and workflows
cd ./.github && mkdir ./tempSchemas || exit 1
curl https://json.schemastore.org/github-action.json --output ./tempSchemas/github-action.json --silent || exit 1
curl https://json.schemastore.org/github-workflow.json --output ./tempSchemas/github-workflow.json --silent || exit 1
downloadSchema 'github-action.json' || exit 1
downloadSchema 'github-workflow.json' || exit 1

# Track exit codes separately so we can run a full validation, report errors, and exit with the correct code
declare EXIT_CODE=0

# This stores all the process IDs of the ajv commands so they can run in parallel
declare ASYNC_PROCESSES

# Arrays of actions and workflows
declare -r ACTIONS=(./actions/*/*/action.yml)
declare -r WORKFLOWS=(./workflows/*.yml)

info 'Validating actions and workflows against their JSON schemas...'

# Validate the actions and workflows using the JSON schemas and ajv https://github.com/ajv-validator/ajv-cli
find ./actions -type f -name "*.yml" -print0 | xargs -0 -I file ajv -s ./tempSchemas/github-action.json -d file --strict=false || EXIT_CODE=1
find ./workflows -type f -name "*.yml" -print0 | xargs -0 -I file ajv -s ./tempSchemas/github-workflow.json -d file --strict=false || EXIT_CODE=1
for ((i=0; i < ${#ACTIONS[@]}; i++)); do
ACTION=${ACTIONS[$i]}
ajv -s ./tempSchemas/github-action.json -d "$ACTION" --strict=false &
ASYNC_PROCESSES[$i]=$!
done

if (( "$EXIT_CODE" != 0 )); then
exit $EXIT_CODE
fi
for ((i=0; i < ${#WORKFLOWS[@]}; i++)); do
WORKFLOW=${WORKFLOWS[$i]}
ajv -s ./tempSchemas/github-workflow.json -d "$WORKFLOW" --strict=false &
ASYNC_PROCESSES[${#ACTIONS[@]} + $i]=$!
done

# Wait for the background builds to finish
for PID in ${ASYNC_PROCESSES[*]}; do
wait $PID
RESULT=$?
if [[ $RESULT != 0 ]]; then
EXIT_CODE=$RESULT
fi
done
Comment on lines 49 to 55

@aldo-expensify aldo-expensify Jul 12, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is what fixed the bug, right?

@roryabraham roryabraham Jul 12, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. When you have an if statement that in-and-of-itself is a command and so it changes the value of $?.


# Cleanup after ourselves and delete the schemas
rm -rf ./tempSchemas

echo
echo 'Lint Github Actions via actionlint (https://github.com/rhysd/actionlint)'
title 'Lint Github Actions via actionlint (https://github.com/rhysd/actionlint)'

# If we are running this on a non-CI machine (e.g. locally), install shellcheck
if [[ -z "${CI}" && -z "$(command -v shellcheck)" ]]; then
Expand All @@ -34,7 +69,15 @@ if [[ -z "${CI}" && -z "$(command -v shellcheck)" ]]; then
brew install shellcheck
fi

curl -s curl https://raw.githubusercontent.com/rhysd/actionlint/main/scripts/download-actionlint.bash | bash -s -- 1.6.13
info 'Downloading actionlint...'
if bash <(curl --silent https://raw.githubusercontent.com/rhysd/actionlint/main/scripts/download-actionlint.bash); then
success 'Successfully downloaded actionlint!'
else
error 'Error downloading actionlint'
exit 1
fi

info 'Linting workflows...'
./actionlint -color || EXIT_CODE=1

# Cleanup after ourselves and delete actionlint
Expand Down