gh-143120: pixi builds for free-threading and TSAN #142872

crusaderky · 2025-12-17T12:08:15Z

Follow-up to #142469

Add pixi infrastructure for

free-threading: compiles with --disable-gil
tsan-free-threading: compiles with --disable-gil --with-thread-sanitizer

Currently, free-threading seems to work well on Linux.
tsan crashes on build: [EDIT read comment below for solution]

[...]/build_tsan ./python -E -S -m sysconfig --generate [...]
 │ │ -posix-vars ;\
[...]
 │ │ Bus error (core dumped)
 │ │ make: *** [Makefile:1162: pybuilddir.txt] Error 135

@kumaraditya303 @lucascolley any idea what's wrong here?

Issue: Add pixi builds for TSan + free-threading to aid downstream development. #143120

crusaderky · 2025-12-17T12:08:35Z

Tools/pixi-packages/freethreading/pixi.toml

copy-paste of asan/pixi.toml

crusaderky · 2025-12-17T12:08:51Z

Tools/pixi-packages/free-threading/recipe.yaml

+  script:
+    file: ../build.sh
+    env:
+      PYTHON_VARIANT: "free-threading"


copy-paste of asan/recipe.yaml, except this one line

crusaderky · 2025-12-17T12:09:04Z

Tools/pixi-packages/tsan-free-threading/pixi.toml

copy-paste of asan/pixi.toml

crusaderky · 2025-12-17T12:09:21Z

Tools/pixi-packages/tsan/recipe.yaml

+  script:
+    file: ../build.sh
+    env:
+      PYTHON_VARIANT: "tsan"


copy-paste of asan/recipe.yaml, except this one line

lucascolley · 2025-12-17T12:10:08Z

@kumaraditya303 @lucascolley any idea what's wrong here?

no clue sorry I haven't seen this before

crusaderky · 2025-12-18T17:23:27Z

The tsan crash is due to the default mmap_rnd_bits on Ubuntu 24.04; it goes away after lowering it:

$ sudo sysctl vm.mmap_rnd_bits
vm.mmap_rnd_bits = 32  # too high
$ sudo sysctl vm.mmap_rnd_bits=28  # reduce it
vm.mmap_rnd_bits = 28

ubuntu-latest github actions workers seems to be unaffected.

hugovk · 2025-12-23T21:56:05Z

There's a lot of copy/paste here, which can make maintenance harder. Does pixi have some concept of code reuse or parametrisation? Maybe not the best example, but something like GitHub Actions' reusable workflows?

ngoldbaum · 2025-12-23T21:58:36Z

@lucascolley is there anything we can do right now to reduce the duplication here? Or does that require fixes in pixi?

lucascolley · 2025-12-23T21:59:35Z

As mentioned in the README already, this is blocked on prefix-dev/pixi#4599.

ngoldbaum

I opened an issue which is needed to make this mergeable. I'm also playing with a NumPy recipe based on this.

ngoldbaum · 2025-12-23T20:34:55Z

Tools/pixi-packages/README.md

- `asan`: ASan-instrumented build with `PYTHON_ASAN=1`
+- `free-threading`
+- `asan`: ASan-instrumented build
+- `tsan`: free-threading, TSan-instrumented build


Should probably be called tsan-free-threading. There should probably also be an asan-free-threading. @lucascolley is there a way to avoid the combinatorial explosion of variants here? Ideally you'd be able to somehow combine these but I think that's probably not tenable at the moment.

I think the answer to this is no because of prefix-dev/pixi#4599
Barring the above, we could write a template + generation script and then commit the output to git? Very ugly if you ask me.

Yes, I would vote for keeping things simple but verbose at the minute until there is a proper solution upstream

Renamed to tsan-free-threading

Tools/pixi-packages/build.sh

lucascolley · 2026-01-06T11:12:49Z

can you try this diff Guido?

diff --git a/Tools/pixi-packages/tsan-free-threading/recipe.yaml b/Tools/pixi-packages/tsan-free-threading/recipe.yaml
index dfae06ad7a5..3417e8b92b1 100644
--- a/Tools/pixi-packages/tsan-free-threading/recipe.yaml
+++ b/Tools/pixi-packages/tsan-free-threading/recipe.yaml
@@ -18,4 +18,6 @@ build:
     env:
       PYTHON_VARIANT: "tsan-free-threading"
+  python:
+    site_packages_path: "lib/python3.13t/site-packages"

 # derived from https://github.com/conda-forge/python-feedstock/blob/main/recipe/meta.yaml

ngoldbaum · 2026-01-06T16:09:42Z

can you try this diff Guido?

Maybe I did something wrong, but I attempted to apply this patch over at ngoldbaum@39df6fe and then updated numpy at ngoldbaum/numpy@d28bebc and it doesn't seem to work, unfortunately, at least on my local setup. CI will test it when a runner picks it up:

https://github.com/numpy/numpy/actions/runs/20754129965/job/59591290742?pr=30510

ngoldbaum · 2026-01-06T16:15:21Z

It looks like the option in the recipe isn't having any effect:

bash-5.3$ ls lib/python3.15/site-packages/
Cython					meson_python-0.18.0.dist-info		mesonpy					pyproject_metadata			tomli
cython-3.2.4.dist-info			meson-1.10.0.dist-info			packaging				pyproject_metadata-0.10.0.dist-info	tomli-2.3.0.dist-info
cython.py				mesonbuild				packaging-25.0.dist-info		pyximport
bash-5.3$ ls lib/python3.15t/site-packages/
pip			pip-25.3.dist-info	README.txt

lucascolley · 2026-01-06T16:43:40Z

I tried it locally and it at least works with pixi global install:

Tools/pixi-packages/tsan-free-threading on 🎋 tsan [🔨?] via 🧚 v0.62.2
❯ pixi global install -e pyt --path ./python-3.15-h60d57d3_0.conda
└── pyt (installed)
    ├─ dependencies: python 3.15
    └─ exposes: idle3, idle3.15, pip3, pip3.15, pydoc3, pydoc3.15, python, python3, python3-config, python3.15, python3.15-config, python3.15t, python3.15t-config

Tools/pixi-packages/tsan-free-threading on 🎋 tsan [🔨?] via 🧚 v0.62.2 took 2s
❯ pixi global install -e pyt cowpy
└── pyt (installed)
    ├─ dependencies: python 3.15, cowpy 1.1.5
    └─ exposes: idle3, idle3.15, pip3, pip3.15, pydoc3, pydoc3.15, python, python3, python3-config, python3.15, python3.15-config, python3.15t, python3.15t-config, cowpy

Tools/pixi-packages/tsan-free-threading on 🎋 tsan [🔨?] via 🧚 v0.62.2
❯ ls -s ~/.pixi/envs/pyt/lib/python3.13t/site-packages/
╭───┬───────────────────────┬──────┬───────┬───────────────╮
│ # │         name          │ type │ size  │   modified    │
├───┼───────────────────────┼──────┼───────┼───────────────┤
│ 0 │ cowpy                 │ dir  │ 128 B │ 2 minutes ago │
│ 1 │ cowpy-1.1.5.dist-info │ dir  │ 320 B │ 2 minutes ago │
╰───┴───────────────────────┴──────┴───────┴───────────────╯

lucascolley · 2026-01-06T16:59:06Z

ah probably that only works for noarch packages and we need something like https://github.com/conda-forge/python-feedstock/blob/c90e3cfe0a44db1b0741359d68c330cd7d861b16/recipe/sitecustomize.py

isuruf · 2026-01-06T19:05:29Z

ah probably that only works for noarch packages and we need something like https://github.com/conda-forge/python-feedstock/blob/c90e3cfe0a44db1b0741359d68c330cd7d861b16/recipe/sitecustomize.py

That workaround is for older conda versions not handling noarch packages properly.

lucascolley · 2026-01-06T19:18:08Z

Hmm yeah on second thought meson-python should be in the right place, let me try to reproduce Nathan's failure

lucascolley · 2026-01-06T20:08:23Z

can you try this diff Guido?

Maybe I did something wrong, but I attempted to apply this patch over at ngoldbaum@39df6fe and then updated numpy at ngoldbaum/numpy@d28bebc and it doesn't seem to work, unfortunately, at least on my local setup. CI will test it when a runner picks it up:

numpy/numpy/actions/runs/20754129965/job/59591290742?pr=30510

doh should have checked the CI log earlier, it is working! Just worked locally for me too

ngoldbaum · 2026-01-06T20:41:02Z

doh should have checked the CI log earlier, it is working! Just worked locally for me too

Oh hey, it worked! That's so strange - I wonder what's different on my setup. It looks like pixi is up-to-date...

lucascolley · 2026-01-06T20:42:16Z

if in doubt try pixi clean, maybe something is being cached that shouldn't be

ngoldbaum · 2026-01-06T20:52:33Z

if in doubt try pixi clean, maybe something is being cached that shouldn't be

Nope, doesn't work. Here's the build log after pixi clean and pixi clean cache: https://gist.github.com/ngoldbaum/8ade90dd534410c6c13e349f23e2d077

The only difference I see compared with the working build on CI is that my build used uv 0.9.22 but the CI build used 0.9.21.

ngoldbaum · 2026-01-08T21:20:36Z

Nope, doesn't work. Here's the build log after pixi clean and pixi clean cache: https://gist.github.com/ngoldbaum/8ade90dd534410c6c13e349f23e2d077

This ended up coming down to there being a .pixi folder in the pixi-packages/tsan-free-threading subfolder of my numpy clone. pixi clean didn't remove it and it took me a while to realize it was actually in a subfolder. Sorry for all the noise on that.

- Remove clang-19 since we are not building the experimental jit - Remove ld_impl because we are not hacking the makefiles - Add site_packages_path - Make recipe.yaml identical with just the variant name difference - Add a generate.sh to generate recipe.yaml from default variant - Add a runtime requirement of libsanitizer for gcc

Tools/pixi-packages/asan/pixi.toml

lucascolley · 2026-01-14T15:35:16Z

looks great, prefix-dev/pixi-build-backends#532 should get us down to a single recipe.yaml, and hopefully in a future Pixi release once we can select a variant/build string we can get down to a single pixi.toml. Seems like the latter is blocked on prefix-dev/pixi#5248.

isuruf · 2026-01-14T17:26:43Z

Are symbolic links allowed? (Especially since these files end up in the python source distribution, it might be bad because of windows). If not we might have to duplicate the files.

lucascolley

could we update the README to point to the new blockers for using a single manifest and recipe per #142872 (comment) ?

bedevere-app bot added the awaiting review label Dec 17, 2025

crusaderky commented Dec 17, 2025

View reviewed changes

Tools/pixi-packages/freethreading/pixi.toml

Copy link

Contributor Author

crusaderky Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy-paste of asan/pixi.toml

crusaderky commented Dec 17, 2025

View reviewed changes

Tools/pixi-packages/tsan-free-threading/pixi.toml Outdated

Copy link

Contributor Author

crusaderky Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy-paste of asan/pixi.toml

crusaderky commented Dec 17, 2025

View reviewed changes

crusaderky marked this pull request as draft December 17, 2025 12:10

bedevere-app bot removed the awaiting review label Dec 17, 2025

crusaderky changed the title ~~WIP pixi builds for free-threading and tsan~~ WIP pixi builds for free-threading and TSAN Dec 17, 2025

This was referenced Dec 23, 2025

Add pixi builds for TSan + free-threading to aid downstream development. #143120

Open

Add note to reduce ASLR randomness for TSan Quansight-Labs/free-threaded-compatibility#304

Merged

hugovk changed the title ~~WIP pixi builds for free-threading and TSAN~~ gh-143120: WIP pixi builds for free-threading and TSAN Dec 23, 2025

hugovk added the skip news label Dec 23, 2025

hugovk added the infra CI, GitHub Actions, buildbots, Dependabot, etc. label Dec 23, 2025

ngoldbaum reviewed Dec 23, 2025

View reviewed changes

ngoldbaum mentioned this pull request Dec 23, 2025

ENH: TSan pixi package definitions numpy/numpy#30510

Draft

crusaderky force-pushed the tsan branch from eb0149f to 242081e Compare January 2, 2026 15:53

crusaderky commented Jan 2, 2026

View reviewed changes

Tools/pixi-packages/build.sh Show resolved Hide resolved

crusaderky added 4 commits January 2, 2026 16:03

pixi builds for free-threading and tsan

d930e12

Rename to tsan-free-threading

57d45b9

Add libclang_rt hack

5ff29b4

Add troubleshooting for tsan

9a2a896

crusaderky force-pushed the tsan branch from b528931 to 9a2a896 Compare January 2, 2026 16:03

crusaderky marked this pull request as ready for review January 2, 2026 16:05

bedevere-app bot added the awaiting review label Jan 2, 2026

crusaderky changed the title ~~gh-143120: WIP pixi builds for free-threading and TSAN~~ gh-143120: pixi builds for free-threading and TSAN Jan 2, 2026

isuruf and others added 9 commits January 8, 2026 14:14

fix typo

456503b

Use single recipe.yaml duplicated across directories

c46f283

use symlinks

d1a38c1

remove generate.sh

57715f5

Merge remote-tracking branch 'origin/main' into tsan

8243c20

Fix

12da3b8

Merge remote-tracking branch 'upstream/main' into tsan

bfa4da3

Fix tsan

1a4106f

lucascolley reviewed Jan 14, 2026

View reviewed changes

Tools/pixi-packages/asan/pixi.toml Outdated Show resolved Hide resolved

switch ABI tag for asan/tsan

2402f77

crusaderky marked this pull request as draft January 14, 2026 15:27

bedevere-app bot removed the awaiting review label Jan 14, 2026

crusaderky added 2 commits January 14, 2026 15:28

git cleanup 1/2

56f4403

git cleanup 2/2

546dcc0

lucascolley reviewed Jan 14, 2026

View reviewed changes

Uh oh!

gh-143120: pixi builds for free-threading and TSAN #142872

Are you sure you want to change the base?

gh-143120: pixi builds for free-threading and TSAN #142872

Uh oh!

Conversation

crusaderky commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucascolley commented Dec 17, 2025

Uh oh!

crusaderky commented Dec 18, 2025

Uh oh!

hugovk commented Dec 23, 2025

Uh oh!

ngoldbaum commented Dec 23, 2025

Uh oh!

lucascolley commented Dec 23, 2025

Uh oh!

ngoldbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lucascolley commented Jan 6, 2026

Uh oh!

ngoldbaum commented Jan 6, 2026

Uh oh!

ngoldbaum commented Jan 6, 2026

Uh oh!

lucascolley commented Jan 6, 2026

Uh oh!

lucascolley commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

isuruf commented Jan 6, 2026

Uh oh!

lucascolley commented Jan 6, 2026

Uh oh!

lucascolley commented Jan 6, 2026

Uh oh!

ngoldbaum commented Jan 6, 2026

Uh oh!

lucascolley commented Jan 6, 2026

Uh oh!

ngoldbaum commented Jan 6, 2026

Uh oh!

ngoldbaum commented Jan 8, 2026

Uh oh!

Uh oh!

lucascolley commented Jan 14, 2026

Uh oh!

isuruf commented Jan 14, 2026

Uh oh!

lucascolley left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

crusaderky commented Dec 17, 2025 •

edited

Loading

lucascolley commented Jan 6, 2026 •

edited

Loading