Skip to content

Commit 9aa8bed

Browse files
jeongseok-metameta-codesync[bot]
authored andcommitted
Add Windows build for PyPI wheels (#1010)
Summary: Add Windows CPU wheel builds to the PyPI publishing workflow, addressing #953. ## Changes ### `pixi.toml` - Add Windows-specific `wheel_build` task using `pip wheel` + pyproject template switching - Add Windows-specific `wheel_repair` task using `delvewheel` to bundle DLLs ### `.github/workflows/publish_to_pypi.yml` - Add `windows-latest` to the `build_cpu_wheels` OS matrix - Add `sccache` for Windows compiler caching (vs `ccache` for macOS) - Add Windows `wheel_repair` step using `delvewheel` - Add Windows CPU test entries (Python 3.12 always, 3.13 on tags/releases) - Platform-specific step conditions using `runner.os` ## Design Decisions 1. **pixi + pip wheel** (not cibuildwheel): Windows doesn't need containerized builds for glibc compatibility. conda-forge provides all dependencies directly, matching the macOS approach. 2. **sccache** for Windows (not ccache): Better compatibility with MSVC, matching the existing `ci_windows.yml` approach. 3. **delvewheel** for DLL bundling: Standard tool for making Windows wheels self-contained (analogous to `auditwheel` on Linux). 4. **CPU only**: GPU builds are not included for Windows yet (current GPU builds use Linux containers via cibuildwheel). Pull Request resolved: #1010 Test Plan: - CI runs for all existing workflows (CI Ubuntu, CI macOS, CI Windows) are triggered - The PyPI Wheels workflow now includes: - `pypi-cpu-py3.12-win64`: Windows CPU wheel build + test - `pypi-cpu-py3.13-win64`: Windows CPU wheel build + test (on tags/releases only) - The `publish_cpu` job automatically picks up Windows wheels via the `wheels-cpu-*` artifact pattern Closes #953 Reviewed By: yutingye Differential Revision: D92630284 Pulled By: jeongseok-meta fbshipit-source-id: 8d7cd101948bf6466d5ead9095dcfde7a3de1cc5
1 parent 112a13d commit 9aa8bed

File tree

6 files changed

+1231
-507
lines changed

6 files changed

+1231
-507
lines changed

.github/workflows/publish_to_pypi.yml

Lines changed: 45 additions & 214 deletions
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,7 @@ jobs:
8484
strategy:
8585
fail-fast: true
8686
matrix:
87-
os: [macos-14]
87+
os: [macos-14, windows-latest]
8888
python-version: ['3.12', '3.13']
8989
include:
9090
- python-version: '3.12'
@@ -93,11 +93,8 @@ jobs:
9393
pixi-environment: py313
9494
- os: macos-14
9595
os-short: mac-arm64
96-
env:
97-
# Enable compiler caching (ccache on Linux/macOS)
98-
CMAKE_C_COMPILER_LAUNCHER: ccache
99-
CMAKE_CXX_COMPILER_LAUNCHER: ccache
100-
96+
- os: windows-latest
97+
os-short: win64
10198
steps:
10299
# Skip Python 3.13 builds on regular commits to reduce CI cost
103100
# Python 3.13 builds will still run when publishing (tags or manual workflow_dispatch with publish=true)
@@ -123,12 +120,18 @@ jobs:
123120
fetch-depth: 0 # Fetch full history for setuptools_scm version detection
124121
fetch-tags: true # Ensure all tags are fetched
125122

126-
- name: Set up ccache
127-
if: steps.should_run.outputs.run == 'true'
123+
# ccache for macOS
124+
- name: Set up ccache (macOS)
125+
if: steps.should_run.outputs.run == 'true' && runner.os == 'macOS'
128126
uses: hendrikmuhs/ccache-action@v1.2
129127
with:
130128
key: ${{ github.workflow }}-${{ matrix.os }}-py${{ matrix.python-version }}
131129

130+
# sccache for Windows
131+
- name: Set up sccache (Windows)
132+
if: steps.should_run.outputs.run == 'true' && runner.os == 'Windows'
133+
uses: mozilla-actions/sccache-action@v0.0.9
134+
132135
- name: Set up pixi
133136
if: steps.should_run.outputs.run == 'true'
134137
uses: prefix-dev/setup-pixi@v0.9.3
@@ -149,176 +152,38 @@ jobs:
149152
- name: Build CPU wheel
150153
if: steps.should_run.outputs.run == 'true'
151154
run: pixi run -e ${{ matrix.pixi-environment }} wheel_build
152-
153-
- name: Print ccache stats
154-
if: steps.should_run.outputs.run == 'true'
155-
run: ccache -s
156-
157-
- name: Upload wheel artifacts
158-
if: steps.should_run.outputs.run == 'true'
159-
uses: actions/upload-artifact@v6
160-
with:
161-
name: wheels-cpu-${{ matrix.os }}-py${{ matrix.python-version }}
162-
path: dist/*.whl
163-
retention-days: 7
164-
165-
build_gpu_wheels:
166-
name: pypi-gpu-py${{ matrix.python-version }}-${{ matrix.os-short }}
167-
runs-on: ${{ matrix.os }}
168-
strategy:
169-
fail-fast: true
170-
# Run GPU builds sequentially to avoid disk space exhaustion
171-
# Each PyTorch+CUDA install needs ~3GB and container overlay shares host disk
172-
max-parallel: 1
173-
matrix:
174-
os: [ubuntu-latest]
175-
python-version: ['3.12', '3.13']
176-
include:
177-
- python-version: '3.12'
178-
pixi-environment: gpu-wheel-build-py312
179-
cuda-version: "12.9.0"
180-
- python-version: '3.13'
181-
pixi-environment: gpu-wheel-build-py313
182-
cuda-version: "12.9.0"
183-
- os: ubuntu-latest
184-
os-short: ubuntu
185-
env:
186-
FULL_CUDA_VERSION: ${{ matrix.cuda-version }}
187-
# Enable compiler caching (ccache on Linux)
188-
CMAKE_C_COMPILER_LAUNCHER: ccache
189-
CMAKE_CXX_COMPILER_LAUNCHER: ccache
190-
191-
steps:
192-
# Skip Python 3.13 builds on regular commits to reduce CI cost
193-
# Python 3.13 builds will still run when publishing (tags or manual workflow_dispatch with publish=true)
194-
- name: Check if should run
195-
id: should_run
196-
run: |
197-
if [[ "${{ matrix.python-version }}" == "3.13" ]]; then
198-
if [[ "${{ startsWith(github.ref, 'refs/tags/v') }}" == "true" ]] || \
199-
[[ "${{ github.event_name == 'workflow_dispatch' && github.event.inputs.publish == 'true' }}" == "true" ]]; then
200-
echo "run=true" >> $GITHUB_OUTPUT
201-
else
202-
echo "run=false" >> $GITHUB_OUTPUT
203-
fi
204-
else
205-
echo "run=true" >> $GITHUB_OUTPUT
206-
fi
207-
shell: bash
208-
209-
- name: Free disk space
210-
if: steps.should_run.outputs.run == 'true' && matrix.os == 'ubuntu-latest'
211-
uses: jlumbroso/free-disk-space@main
212-
with:
213-
# Remove all default tools and applications
214-
tool-cache: false # Keep tool cache for Python
215-
android: true
216-
dotnet: true
217-
haskell: true
218-
large-packages: true
219-
docker-images: false # Keep container images for cibuildwheel
220-
swap-storage: true
221-
222-
- name: Maximize build space (Ubuntu only)
223-
# DISABLED: This action restructures the filesystem and breaks container engine functionality
224-
# The "Free disk space" step above provides enough space for the build
225-
if: false && steps.should_run.outputs.run == 'true' && matrix.os == 'ubuntu-latest'
226-
uses: easimon/maximize-build-space@master
227-
with:
228-
root-reserve-mb: 30720
229-
swap-size-mb: 1024
230-
remove-dotnet: true
231-
remove-android: true
232-
remove-haskell: true
233-
remove-codeql: true
234-
remove-docker-images: false # Keep container images for cibuildwheel
235-
236-
- uses: actions/checkout@v6
237-
if: steps.should_run.outputs.run == 'true'
238-
with:
239-
submodules: recursive
240-
fetch-depth: 0 # Fetch full history for setuptools_scm version detection
241-
fetch-tags: true # Ensure all tags are fetched
242-
243-
- name: Set up ccache
244-
if: steps.should_run.outputs.run == 'true'
245-
uses: hendrikmuhs/ccache-action@v1.2
246-
with:
247-
key: ${{ github.workflow }}-${{ matrix.os }}-py${{ matrix.python-version }}-gpu
248-
249-
- name: Set up pixi
250-
if: steps.should_run.outputs.run == 'true'
251-
uses: prefix-dev/setup-pixi@v0.9.3
252-
with:
253-
pixi-version: latest
254-
cache: true
255-
cache-write: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
256-
environments: ${{ matrix.pixi-environment }}
257-
258-
- name: Generate pyproject configs
259-
if: steps.should_run.outputs.run == 'true'
260-
run: pixi run -e ${{ matrix.pixi-environment }} generate_pyproject
261-
262-
- name: Clean distribution artifacts
263-
if: steps.should_run.outputs.run == 'true'
264-
run: pixi run -e ${{ matrix.pixi-environment }} wheel_clean
265-
266-
- name: Determine version for container builds
267-
if: steps.should_run.outputs.run == 'true'
268-
id: get_version
269-
run: |
270-
# Install setuptools_scm to determine version from git tags
271-
pip install setuptools_scm
272-
# Get the version that setuptools_scm would generate
273-
FULL_VERSION=$(python -c "from setuptools_scm import get_version; print(get_version())")
274-
echo "Detected version: $FULL_VERSION"
275-
# Strip local version part (e.g., +gad8997da8) - PyPI doesn't allow local versions
276-
# This converts "0.1.101.dev26+gad8997da8" to "0.1.101.dev26"
277-
VERSION=$(echo "$FULL_VERSION" | sed 's/+.*//')
278-
echo "Version for PyPI: $VERSION"
279-
echo "version=$VERSION" >> $GITHUB_OUTPUT
280-
281-
- name: Build GPU wheel
282-
if: steps.should_run.outputs.run == 'true'
283-
timeout-minutes: 60
284155
env:
285-
# Pass version to cibuildwheel so containers can use it
286-
# Without this, setuptools_scm inside the container would generate 0.0.post1
287-
CIBW_ENVIRONMENT_PASS_LINUX: SETUPTOOLS_SCM_PRETEND_VERSION
288-
SETUPTOOLS_SCM_PRETEND_VERSION: ${{ steps.get_version.outputs.version }}
289-
run: |
290-
echo "=== Starting GPU wheel build ==="
291-
echo "Environment: ${{ matrix.pixi-environment }}"
292-
echo "Version: $SETUPTOOLS_SCM_PRETEND_VERSION"
293-
echo "=== Detecting container engine ==="
294-
source scripts/detect_container_engine.sh
295-
$CIBW_CONTAINER_ENGINE --version || echo "Container engine not available"
296-
$CIBW_CONTAINER_ENGINE info || echo "Container engine info failed"
297-
echo "=== Running wheel_build ==="
298-
pixi run -e ${{ matrix.pixi-environment }} wheel_build
299-
echo "=== Build completed ==="
300-
301-
- name: Repair GPU wheel (Linux only)
302-
if: steps.should_run.outputs.run == 'true' && matrix.os == 'ubuntu-latest'
156+
# Enable compiler caching: ccache on macOS, sccache on Windows
157+
CMAKE_C_COMPILER_LAUNCHER: ${{ runner.os == 'Windows' && 'sccache' || 'ccache' }}
158+
CMAKE_CXX_COMPILER_LAUNCHER: ${{ runner.os == 'Windows' && 'sccache' || 'ccache' }}
159+
SCCACHE_GHA_ENABLED: ${{ runner.os == 'Windows' && 'true' || '' }}
160+
161+
# Repair wheel on Windows using delvewheel (bundles DLLs)
162+
- name: Repair wheel (Windows)
163+
if: steps.should_run.outputs.run == 'true' && runner.os == 'Windows'
303164
run: pixi run -e ${{ matrix.pixi-environment }} wheel_repair
304165

305-
- name: Print ccache stats
306-
if: steps.should_run.outputs.run == 'true'
166+
- name: Print ccache stats (macOS)
167+
if: steps.should_run.outputs.run == 'true' && runner.os == 'macOS'
307168
run: ccache -s
308169

170+
- name: Print sccache stats (Windows)
171+
if: steps.should_run.outputs.run == 'true' && runner.os == 'Windows'
172+
run: sccache --show-stats
173+
309174
- name: Upload wheel artifacts
310175
if: steps.should_run.outputs.run == 'true'
311176
uses: actions/upload-artifact@v6
312177
with:
313-
name: wheels-gpu-${{ matrix.os }}-py${{ matrix.python-version }}
178+
name: wheels-cpu-${{ matrix.os }}-py${{ matrix.python-version }}
314179
path: dist/*.whl
315180
retention-days: 7
316181

317182
# Regression test: Verify pip wheels work correctly (import test, parallel operations)
318183
# Tests all build variants with same skip logic as builds (py3.13 only on tags/releases)
319184
test_pip_wheels:
320185
name: Test pip wheel - ${{ matrix.variant }}-py${{ matrix.python-version }}-${{ matrix.os-short }}
321-
needs: [build_cpu_wheels_linux, build_cpu_wheels, build_gpu_wheels]
186+
needs: [build_cpu_wheels_linux, build_cpu_wheels]
322187
runs-on: ${{ matrix.os }}
323188
strategy:
324189
fail-fast: false
@@ -356,22 +221,22 @@ jobs:
356221
pixi-environment: py313
357222
artifact-pattern: wheels-cpu-macos-14-py3.13
358223
wheel-type: cpu
359-
# Linux GPU - always test py3.12 (import-only, no GPU required)
360-
- os: ubuntu-latest
361-
os-short: linux
362-
variant: gpu
224+
# Windows - always test py3.12
225+
- os: windows-latest
226+
os-short: win64
227+
variant: cpu
363228
python-version: '3.12'
364229
pixi-environment: py312
365-
artifact-pattern: wheels-gpu-ubuntu-latest-py3.12
366-
wheel-type: gpu
367-
# Linux GPU - py3.13 only on tags/releases
368-
- os: ubuntu-latest
369-
os-short: linux
370-
variant: gpu
230+
artifact-pattern: wheels-cpu-windows-latest-py3.12
231+
wheel-type: cpu
232+
# Windows - py3.13 only on tags/releases
233+
- os: windows-latest
234+
os-short: win64
235+
variant: cpu
371236
python-version: '3.13'
372237
pixi-environment: py313
373-
artifact-pattern: wheels-gpu-ubuntu-latest-py3.13
374-
wheel-type: gpu
238+
artifact-pattern: wheels-cpu-windows-latest-py3.13
239+
wheel-type: cpu
375240

376241
steps:
377242
# Skip py3.13 tests on regular commits (same logic as builds)
@@ -415,13 +280,17 @@ jobs:
415280
- name: List downloaded wheels
416281
if: steps.should_run.outputs.run == 'true'
417282
run: ls -lh dist/
283+
shell: bash
418284

419-
- name: Test wheel with uv (pixi wheel_test)
285+
# Run test_wheel.py directly to test pre-built wheel artifacts
286+
# We don't use `pixi run wheel_test` because it has a depends-on: wheel_build
287+
# which would rebuild the wheel instead of testing the downloaded artifact
288+
- name: Test wheel with uv
420289
if: steps.should_run.outputs.run == 'true'
421290
env:
422291
WHEEL_TEST_PYTHON_VERSION: cp${{ matrix.python-version == '3.12' && '312' || '313' }}
423292
WHEEL_TEST_TYPE: ${{ matrix.wheel-type }}
424-
run: pixi run -e ${{ matrix.pixi-environment }} wheel_test
293+
run: pixi run -e ${{ matrix.pixi-environment }} python scripts/test_wheel.py
425294

426295
publish_cpu:
427296
name: Publish pymomentum-cpu to PyPI
@@ -460,41 +329,3 @@ jobs:
460329
(github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')) ||
461330
(github.event_name == 'workflow_dispatch' && github.event.inputs.publish == 'true' && github.event.inputs.test_pypi != 'true')
462331
uses: pypa/gh-action-pypi-publish@release/v1
463-
464-
publish_gpu:
465-
name: Publish pymomentum-gpu to PyPI
466-
needs: [build_gpu_wheels, test_pip_wheels]
467-
runs-on: ubuntu-latest
468-
environment:
469-
name: pypi-gpu
470-
url: https://pypi.org/p/pymomentum-gpu
471-
permissions:
472-
id-token: write # IMPORTANT: mandatory for trusted publishing
473-
# Only publish on tag push or manual workflow dispatch with publish=true
474-
if: |
475-
(github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')) ||
476-
(github.event_name == 'workflow_dispatch' && github.event.inputs.publish == 'true')
477-
478-
steps:
479-
- name: Download GPU wheel artifacts
480-
uses: actions/download-artifact@v7
481-
with:
482-
path: dist
483-
pattern: wheels-gpu-*
484-
merge-multiple: true
485-
486-
- name: List GPU distributions
487-
run: ls -lh dist/
488-
489-
- name: Publish GPU to TestPyPI
490-
if: |
491-
github.event_name == 'workflow_dispatch' && github.event.inputs.publish == 'true' && github.event.inputs.test_pypi == 'true'
492-
uses: pypa/gh-action-pypi-publish@release/v1
493-
with:
494-
repository-url: https://test.pypi.org/legacy/
495-
496-
- name: Publish GPU to PyPI
497-
if: |
498-
(github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')) ||
499-
(github.event_name == 'workflow_dispatch' && github.event.inputs.publish == 'true' && github.event.inputs.test_pypi != 'true')
500-
uses: pypa/gh-action-pypi-publish@release/v1

0 commit comments

Comments
 (0)