Skip to content

GH-49057: [Python][Packaging] Remove test base docker image and use single windows base image for both build and test containers #48996

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
raulcd wants to merge 13 commits into
base: main
Choose a base branch
from
Open

GH-49057: [Python][Packaging] Remove test base docker image and use single windows base image for both build and test containers #48996

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
ac76f96
WIP: [Python] Simplify Windows wheel build ecosystem
raulcd Jan 26, 2026
054b8dc
Add python_variant_suffix for testing free-threaded
raulcd Jan 27, 2026
a07afbf
Try upgrading wheel windows image revision
raulcd Jan 27, 2026
e1ab7da
Fix base image
raulcd Jan 28, 2026
6665098
Remove unnecessary change
raulcd Jan 28, 2026
2aa3a1f
Base for testing images should be the base image not the build image
raulcd Jan 28, 2026
1ca2938
Dockerfiles for test should specify PYTHON_CMD
raulcd Jan 28, 2026
f4d33c2
Use pymanager to install Python for GCS testbench
raulcd Jan 29, 2026
eac026f
Try removing python installation and see if there's a python on the b…
raulcd Jan 30, 2026
98410dd
Use base image Python instead for gcs
raulcd Jan 30, 2026
a1b0652
Revert "Use base image Python instead for gcs"
raulcd Feb 3, 2026
e4058ef
Revert "Try removing python installation and see if there's a python …
raulcd Feb 3, 2026
796f7c7
Revert "Use pymanager to install Python for GCS testbench"
raulcd Feb 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,8 @@ VCPKG="4334d8b4c8916018600212ab4dd4bbdc343065d1" # 2025.09.17 Release
# ci/docker/python-*-windows-*.dockerfile or the vcpkg config.
# This is a workaround for our CI problem that "archery docker build" doesn't
# use pulled built images in dev/tasks/python-wheels/github.windows.yml.
PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2026-01-22
PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2026-01-22
PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2026-01-27
PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2026-01-27

# Use conanio/${CONAN_BASE}:{CONAN_VERSION} for "docker compose run --rm conan".
# See https://github.com/conan-io/conan-docker-tools#readme and
Expand Down
18 changes: 6 additions & 12 deletions ci/docker/python-free-threaded-wheel-windows-test-vs2022.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,18 +26,12 @@ FROM ${base}

ARG python=3.13

# hadolint ignore=SC1072
RUN (if "%python%"=="3.13" setx PYTHON_VERSION "3.13.1") & \
(if "%python%"=="3.14" setx PYTHON_VERSION "3.14.0")

SHELL ["powershell", "-NoProfile", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN $version = $env:PYTHON_VERSION; \
$filename = 'python-' + $version + '-amd64.exe'; \
$url = 'https://www.python.org/ftp/python/' + $version + '/' + $filename; \
Invoke-WebRequest -Uri $url -OutFile $filename; \
Start-Process -FilePath $filename -ArgumentList '/quiet', 'Include_freethreaded=1' -Wait

ENV PYTHON_CMD="py -${python}t"
ARG python_variant_suffix=""
ENV PYTHON_VERSION=${python}${python_variant_suffix}

RUN pymanager install %PYTHON_VERSION%

ENV PYTHON_CMD="py -${python}${python_variant_suffix}"

SHELL ["cmd", "/S", "/C"]
RUN %PYTHON_CMD% -m pip install -U pip setuptools & \
Expand Down
65 changes: 0 additions & 65 deletions ci/docker/python-wheel-windows-test-vs2022-base.dockerfile
View file
Open in desktop

This file was deleted.

15 changes: 7 additions & 8 deletions ci/docker/python-wheel-windows-test-vs2022.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,15 @@ FROM ${base}

# hadolint shell=cmd.exe

# Define the full version number otherwise choco falls back to patch number 0 (3.10 => 3.10.0)
ARG python=3.10
RUN (if "%python%"=="3.10" setx PYTHON_VERSION "3.10.11" && setx PYTHON_CMD "py -3.10") & \
(if "%python%"=="3.11" setx PYTHON_VERSION "3.11.9" && setx PYTHON_CMD "py -3.11") & \
(if "%python%"=="3.12" setx PYTHON_VERSION "3.12.10" && setx PYTHON_CMD "py -3.12") & \
(if "%python%"=="3.13" setx PYTHON_VERSION "3.13.9" && setx PYTHON_CMD "py -3.13") & \
(if "%python%"=="3.14" setx PYTHON_VERSION "3.14.0" && setx PYTHON_CMD "py -3.14")

# hadolint ignore=DL3059
RUN choco install -r -y --pre --no-progress --force python --version=%PYTHON_VERSION%
ARG python_variant_suffix=""
ENV PYTHON_VERSION=${python}${python_variant_suffix}

RUN pymanager install %PYTHON_VERSION%

ENV PYTHON_CMD="py -${python}${python_variant_suffix}"

# hadolint ignore=DL3059
RUN %PYTHON_CMD% -m pip install -U pip setuptools

Expand Down
15 changes: 15 additions & 0 deletions ci/docker/python-wheel-windows-vs2022-base.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,21 @@ RUN choco install --no-progress -r -y git gzip ninja wget
# Add UNIX tools to PATH
RUN setx path "%path%;C:\Program Files\Git\usr\bin"

# Install git, wget, minio
RUN choco install --no-progress -r -y git wget
RUN curl https://dl.min.io/server/minio/release/windows-amd64/archive/minio.RELEASE.2025-01-20T14-49-07Z `
--output "C:\Windows\Minio.exe"

# Install the GCS testbench using a well-known Python version.
# NOTE: cannot use pipx's `--fetch-missing-python` because of
# https://github.com/pypa/pipx/issues/1521, therefore download Python ourselves.
Copy link
Member

@pitrou pitrou Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems the pipx issue was solved, should we reconsider this?

Copy link
Member

@pitrou pitrou Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or we can use pymanager instead of choco, I don't know. (but this might be fine anyway)

Copy link
Member Author

@raulcd raulcd Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We still would require to install python manually (either via choco or pymanager) to pip install pipx, right? As the base image won't have any python by default. I don't understand what the --fetch-missing-python problem was trying to solve on that specific case.
I am fine trying to install Python here with pymanager instead of choco.

Copy link
Member

@pitrou pitrou Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I think the image had some Python installed before. If choco works fine then we can probably keep it for now...

RUN choco install -r -y --pre --no-progress python --version=3.11.9
ENV PIPX_BIN_DIR=C:\\Windows\\
ENV PIPX_PYTHON="C:\Python311\python.exe"
COPY ci/scripts/install_gcs_testbench.bat C:/arrow/ci/scripts/
RUN call "C:\arrow\ci\scripts\install_gcs_testbench.bat" && `
storage-testbench -h

# Install vcpkg
#
# Compiling vcpkg itself from a git tag doesn't work anymore since vcpkg has
Expand Down
1 change: 0 additions & 1 deletion ci/docker/python-wheel-windows-vs2022.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
ARG base
FROM ${base}

# Define the full version number otherwise choco falls back to patch number 0 (3.10 => 3.10.0)
ARG python=3.10

ARG python_variant_suffix=""
Expand Down
13 changes: 3 additions & 10 deletions compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,6 @@ x-hierarchy:
- python-wheel-windows-vs2022-base:
- python-wheel-windows-vs2022
- python-free-threaded-wheel-windows-vs2022
- python-wheel-windows-test-base:
- python-wheel-windows-test
- python-free-threaded-wheel-windows-test

Expand Down Expand Up @@ -1416,18 +1415,11 @@ services:
volumes: *python-wheel-windows-vs2022-volumes
command: arrow\\ci\\scripts\\python_wheel_windows_build.bat

python-wheel-windows-test-base:
image: ${REPO}:python-wheel-windows-test-vs2022-base-${PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION}
build:
context: .
dockerfile: ci/docker/python-wheel-windows-test-vs2022-base.dockerfile
volumes: *python-wheel-windows-vs2022-volumes

python-wheel-windows-test:
image: ${REPO}:python-${PYTHON}-wheel-windows-test-vs2022-${PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION}
build:
args:
base: ${REPO}:python-wheel-windows-test-vs2022-base-${PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION}
base: ${REPO}:python-wheel-windows-vs2022-base-vcpkg-${VCPKG}-${PYTHON_WHEEL_WINDOWS_IMAGE_REVISION}
python: ${PYTHON}
context: .
dockerfile: ci/docker/python-wheel-windows-test-vs2022.dockerfile
Expand All @@ -1438,8 +1430,9 @@ services:
image: ${REPO}:python-${PYTHON}-free-threaded-wheel-windows-test-vs2022-${PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION}
build:
args:
base: ${REPO}:python-wheel-windows-test-vs2022-base-${PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION}
base: ${REPO}:python-wheel-windows-vs2022-base-vcpkg-${VCPKG}-${PYTHON_WHEEL_WINDOWS_IMAGE_REVISION}
python: ${PYTHON}
python_variant_suffix: t
context: .
dockerfile: ci/docker/python-free-threaded-wheel-windows-test-vs2022.dockerfile
volumes: *python-wheel-windows-vs2022-volumes
Expand Down