Skip to content

Unify build workflows #16096

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Brett Cannon (brettcannon) merged 8 commits into from
May 3, 2021
Merged

Unify build workflows #16096

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
7ff3fa3
Create a unified 'build' workflow
brettcannon Apr 29, 2021
18f1916
Test new setup
brettcannon Apr 29, 2021
263172a
Try to fix using env to specify the Python version
brettcannon Apr 29, 2021
4e88980
Try to use an environment variable in the test matrix
brettcannon Apr 29, 2021
4fdc73d
Use 3.x as the newest version of Python to support
brettcannon Apr 29, 2021
1158ced
Test using setup outputs
brettcannon Apr 29, 2021
2a0851e
Final touch-ups for the PR
brettcannon Apr 29, 2021
9b6506f
Drop old workflow files
brettcannon Apr 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 69 additions & 50 deletions .github/workflows/insiders.yml → .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,27 +1,49 @@
name: Insiders Build
name: Build

on:
push:
branches:
- main
- 'main'
- 'release'
- 'release/*'
- 'release-*'

env:
PYTHON_DEBUGPY_WHEEL_VERSION: 3.8
NODE_VERSION: 12.15.0
PYTHON_VERSION: 3.9
MOCHA_REPORTER_JUNIT: true # Use the mocha-multi-reporters and send output to both console (spec) and JUnit (mocha-junit-reporter). Also enables a reporter which exits the process running the tests if it haven't already.

@karrtikr Kartik Raj (karrtikr) May 4, 2021

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MOCHA_REPORTER_JUNIT enables a reporter which exits the process running the tests if it haven't already. It had been observed that even though all the tests pass, for some reason the action did not exit and got stuck. I think we should restore it. Brett Cannon (@brettcannon)

@brettcannon Brett Cannon (brettcannon) May 5, 2021

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/microsoft/vscode-python/blob/main/.github/workflows/build.yml

Also feel free to open a PR for this sort of thing. I also left it out since it was false in 'release' but true in 'insiders'. I have just made it true regardless of what build is occurring.

@karrtikr Kartik Raj (karrtikr) May 5, 2021

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see, understood 👍

ARTIFACT_NAME_VSIX: ms-python-insiders-vsix
VSIX_NAME: ms-python-insiders.vsix
TEST_RESULTS_DIRECTORY: .
# Force a path with spaces and to test extension works in these scenarios
# Unicode characters are causing 2.7 failures so skip that for now.
special-working-directory: './path with spaces'
special-working-directory-relative: 'path with spaces'

jobs:
setup:
name: Set up
if: github.repository == 'microsoft/vscode-python'
runs-on: ubuntu-latest
defaults:
run:
shell: python
outputs:
vsix_name: ${{ steps.vsix_names.outputs.vsix_name }}
vsix_artifact_name: $ {{ steps.vsix_names.outputs.vsix_artifact_name }}
steps:
- name: VSIX names
id: vsix_names
run: |
import os
if os.environ["GITHUB_REF"].endswith("/main"):
vsix_type = "insiders"
else:
vsix_type = "release"
print(f"::set-output name=vsix_name::ms-python-{vsix_type}.vsix")
print(f"::set-output name=vsix_artifact_name::ms-python-{vsix_type}-vsix")

build-vsix:
name: Build VSIX
runs-on: ubuntu-latest
if: github.repository == 'microsoft/vscode-python'
needs: setup
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
Expand All @@ -31,10 +53,10 @@ jobs:
with:
node-version: ${{env.NODE_VERSION}}

- name: Use Python ${{env.PYTHON_VERSION}}
- name: Use Python 3.x
uses: actions/setup-python@v2
with:
python-version: ${{env.PYTHON_VERSION}}
python-version: 3.x

- name: Upgrade pip
run: python -m pip install -U pip
Expand All @@ -44,19 +66,19 @@ jobs:
id: build-vsix

- name: Rename VSIX
if: steps.build-vsix.outputs.path != env.VSIX_NAME
run: mv ${{ steps.build-vsix.outputs.path }} ${{ env.VSIX_NAME }}
if: steps.build-vsix.outputs.path != needs.setup.outputs.vsix_name
run: mv ${{ steps.build-vsix.outputs.path }} ${{ needs.setup.outputs.vsix_name }}

- uses: actions/upload-artifact@v2
with:
name: ${{env.ARTIFACT_NAME_VSIX}}
path: ${{env.VSIX_NAME}}
retention-days: 7
name: ${{ needs.setup.outputs.vsix_artifact_name }}
path: ${{ needs.setup.outputs.vsix_name }}
retention-days: 14

lint:
name: Lint
runs-on: ubuntu-latest
if: github.repository == 'microsoft/vscode-python'
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
Expand All @@ -81,10 +103,10 @@ jobs:
- name: Run formatting on TypeScript code
run: npm run format-check

- name: Use Python ${{env.PYTHON_VERSION}}
- name: Use Python 3.x
uses: actions/setup-python@v2
with:
python-version: ${{env.PYTHON_VERSION}}
python-version: 3.x
Comment thread
brettcannon marked this conversation as resolved.

- name: Run Black on Python code
run: |
Expand All @@ -95,20 +117,19 @@ jobs:
### Non-smoke tests
tests:
name: Tests
# The value of runs-on is the OS of the current job (specified in the strategy matrix below) instead of being hardcoded.
if: github.repository == 'microsoft/vscode-python'
runs-on: ${{ matrix.os }}
defaults:
run:
working-directory: ${{env.special-working-directory}}
if: github.repository == 'microsoft/vscode-python'
strategy:
fail-fast: false
matrix:
# We're not running CI on macOS for now because it's one less matrix entry to lower the number of runners used,
# macOS runners are expensive, and we assume that Ubuntu is enough to cover the UNIX case.
# We're not running CI on macOS for now because it's one less matrix
# entry to lower the number of runners used, macOS runners are expensive,
# and we assume that Ubuntu is enough to cover the UNIX case.
os: [ubuntu-latest, windows-latest]
# Run the tests on the oldest and most recent versions of Python.
python: [2.7, 3.9]
python: ['2.7', '3.x']
test-suite: [ts-unit, python-unit, venv, single-workspace, multi-workspace, debugger, functional]
steps:
- name: Checkout
Expand All @@ -127,7 +148,13 @@ jobs:
- name: Compile
run: npx gulp prePublishNonBundle

- name: Use Python ${{matrix.python}}
- name: Install Python ${{ env.PYTHON_DEBUGPY_WHEEL_VERSION }} for debugpy wheels
if: matrix.test_suite == 'debugger'
uses: actions/setup-python@v2
with:
python-version: ${{ env.PYTHON_DEBUGPY_WHEEL_VERSION }}

- name: Install Python ${{matrix.python}}
uses: actions/setup-python@v2
with:
python-version: ${{matrix.python}}
Expand All @@ -139,7 +166,7 @@ jobs:

- name: Install Python requirements
run: |
python -m pip --disable-pip-version-check install -t ./pythonFiles/lib/python --no-cache-dir --implementation py --no-deps --upgrade -r requirements.txt --no-user
python -m pip --disable-pip-version-check install -t ./pythonFiles/lib/python --no-cache-dir --implementation py --no-deps --upgrade -r requirements.txt
# We need to have debugpy so that tests relying on it keep passing, but we don't need install_debugpy's logic in the test phase.
python -m pip --disable-pip-version-check install -t ./pythonFiles/lib/python --no-cache-dir --implementation py --no-deps --upgrade --pre debugpy

Expand All @@ -148,11 +175,11 @@ jobs:

- name: Install debugpy wheels (python 3.8)
run: |
python -m pip install wheel
python -m pip --disable-pip-version-check install -r build/debugger-install-requirements.txt
python ./pythonFiles/install_debugpy.py
python${{ env.PYTHON_DEBUGPY_WHEEL_VERSION }} -m pip install wheel
python${{ env.PYTHON_DEBUGPY_WHEEL_VERSION }} -m pip --disable-pip-version-check install -r build/debugger-install-requirements.txt
python${{ env.PYTHON_DEBUGPY_WHEEL_VERSION }} ./pythonFiles/install_debugpy.py
shell: bash
if: matrix.test-suite == 'debugger' && matrix.python == 3.8
if: matrix.test-suite == 'debugger'

- name: Install debugpy (python 2.7)
run: |
Expand Down Expand Up @@ -247,12 +274,10 @@ jobs:
# Run TypeScript unit tests only for Python 3.X.
- name: Run TypeScript unit tests
run: npm run test:unittests
if: matrix.test-suite == 'ts-unit' && startsWith(matrix.python, 3.)
if: matrix.test-suite == 'ts-unit' && startsWith(matrix.python, '3.')

# Run the Python tests in our codebase.
- name: Run Python unit tests
run: |
python pythonFiles/tests/run_all.py
run: python pythonFiles/tests/run_all.py
if: matrix.test-suite == 'python-unit'

# The virtual environment based tests use the `testSingleWorkspace` set of tests
Expand Down Expand Up @@ -305,20 +330,17 @@ jobs:

smoke-tests:
name: Smoke tests
# The value of runs-on is the OS of the current job (specified in the strategy matrix below) instead of being hardcoded.
runs-on: ${{ matrix.os }}
if: github.repository == 'microsoft/vscode-python'
needs: [build-vsix]
runs-on: ${{ matrix.os }}
needs: [setup, build-vsix]
strategy:
fail-fast: false
matrix:
# We're not running CI on macOS for now because it's one less matrix entry to lower the number of runners used,
# macOS runners are expensive, and we assume that Ubuntu is enough to cover the UNIX case.
os: [ubuntu-latest, windows-latest]
# 3.8 is still required here so that jupyter can install.
python: [3.8]
python: ['3.x']
steps:
# Need the source to have the tests available.
- name: Checkout
uses: actions/checkout@v2

Expand Down Expand Up @@ -351,9 +373,8 @@ jobs:
- name: Download VSIX
uses: actions/download-artifact@v2
with:
name: ${{env.ARTIFACT_NAME_VSIX}}
name: ${{ needs.setup.outputs.vsix_artifactname }}

# Compile the test files.
- name: Prepare for smoke tests
run: npx tsc -p ./
shell: bash
Expand All @@ -374,18 +395,16 @@ jobs:

insider-tests:
name: Insider tests
# The value of runs-on is the OS of the current job (specified in the strategy matrix below) instead of being hardcoded.
runs-on: ${{ matrix.os }}
if: github.repository == 'microsoft/vscode-python'
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
# We're not running CI on macOS for now because it's one less matrix entry to lower the number of runners used,
# macOS runners are expensive, and we assume that Ubuntu is enough to cover the UNIX case.
os: [ubuntu-latest]
python: [3.8]
python: [3.8] # For Jupyter.
steps:
# Need the source to have the tests available.
- name: Checkout
uses: actions/checkout@v2

Expand Down Expand Up @@ -439,22 +458,22 @@ jobs:

upload:
name: Upload VSIX to Azure Blob Storage
if: github.repository == 'microsoft/vscode-python'
if: github.repository == 'microsoft/vscode-python' && github.ref == 'refs/heads/main'
needs: [setup, tests, smoke-tests, build-vsix]
runs-on: ubuntu-latest
needs: [tests, smoke-tests, build-vsix]
env:
BLOB_CONTAINER_NAME: extension-builds
BLOB_NAME: ms-python-insiders.vsix
steps:
- name: Download VSIX
uses: actions/download-artifact@v2
with:
name: ${{ env.ARTIFACT_NAME_VSIX }}
name: ${{ needs.setup.outputs.vsix_artifact_name }}
- name: Azure Login
uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
- name: Upload to Blob Storage
run: az storage blob upload --file ${{ env.VSIX_NAME }} --account-name pvsc --container-name ${{ env.BLOB_CONTAINER_NAME }} --name ${{ env.BLOB_NAME }} --auth-mode login
run: az storage blob upload --file ${{ needs.setup.outputs.vsix_name }} --account-name pvsc --container-name ${{ env.BLOB_CONTAINER_NAME }} --name ${{ env.BLOB_NAME }} --auth-mode login
- name: Get URL to uploaded VSIX
run: az storage blob url --account-name pvsc --container-name ${{ env.BLOB_CONTAINER_NAME }} --name ${{ env.BLOB_NAME }} --auth-mode login
Loading