`build-details.json` v1.1

[FFY00]

Issue Description

This issue is meant to track changes we'd like to make in a new build-details.json schema version.

• Allow libpython.link_extensions to be specified when libpython.dynamic is missing
  • Use case: The Python installation may not ship a library if the extensions aren't meant to link against it (Prototype support for PEP 739 mesonbuild/meson#13945 (comment)).
• ...

Code of Conduct

• I am aware that participants in this repository must follow the PSF Code of Conduct.

[mgorny]

• Allow libpython.link_extensions to be specified when libpython.dynamic is missing

  • Use case: The Python installation may not ship a library if the extensions aren't meant to link against it (

This is an interesting problem. To elaborate a bit, PyPy3 can be built either with a shared libpypy3.x-c.so or without. In Gentoo, we've decided to go for the latter since we're splitting the packages a bit, and avoiding the shared library made versioning easier for us. Perhaps we should finally revisit this decision, given it's causing issues here and there, but I think it's still a valid point to consider.

That said, while I think it makes sense to allow specifying it, I think it would also make sense to add a "default" that if libpython.dynamic is missing, then libpython.link_extensions is implicitly false.

[FFY00]

That said, while I think it makes sense to allow specifying it, I think it would also make sense to add a "default" that if libpython.dynamic is missing, then libpython.link_extensions is implicitly false.

Humm, I'd prefer the opposite, or more accurately, for its absence to mean that extensions aren't supported. Defaulting to False could possibly lead to situations where we are building extensions for an installation that doesn't support them. Defaulting to not compiling seems more robust. Even compiling but trying to link against libpython, as the linking step would likely fail.

That said, this is hidden behind the libpython key anyway, so it doesn't matter that much, as to avoid these situations you would simply need to ensure you ship a correct build-details.json file. Though, I still think the opposite default would be more robust.

[mgorny]

Another idea: the pkgconfig "package name" (filename).

Right now the spec is slightly inconsistent. On one hand, it provides exact library and include paths that can be used as-is. On the other, it provides only the pkg-config directory but expects the caller to figure out the correct pkg-config package name / filename.

Something like:

  "c_api": {
    "headers": "/tmp/z/include/python3.15",
    "pkgconfig_path": "/tmp/z/lib/pkgconfig",
    "pkgconfig_name": "python-3.15"
  }

(not sure if we should have separate entry for "embed" or just infer it)

[mgorny]

Another problem: abi.stable_abi_suffix on PyPy.

The current demo logic simply doesn't include the relevant key in build-details.json since there is no suffix matching .abi in suffixes used by PyPy:

        # EXTENSION_SUFFIXES has been constant for a long time, and currently we
        # don't have a better information source to find the  stable ABI suffix.
        for suffix in importlib.machinery.EXTENSION_SUFFIXES:
            if suffix.startswith('.abi'):
                data['abi']['stable_abi_suffix'] = suffix
                break

https://github.com/FFY00/python-introspection/blob/8b2e1299d4e2b1e4d2a82a33a2fd7f0e82b3f52c/python_introspection/scripts/generate-build-details.py#L96-L101

However, for example (without PEP 739) Meson itself treats regular extension suffix as "limited API suffix":

# pypy supports modules targeting the limited api but
# does not use a special suffix to distinguish them:
# https://doc.pypy.org/en/latest/cpython_differences.html#permitted-abi-tags-in-extensions
if is_pypy:
    limited_api_suffix = suffix

https://github.com/mesonbuild/meson/blob/555502761e5bb78b0a5431f9c46987f0413fbe44/mesonbuild/scripts/python_info.py#L103-L107

To be honest, I don't know which approach is more correct: i.e. if we should include abi.stable_abi_suffix in PyPy or if Meson should have a special fallback for PyPy when the key is missing.

The current wording of the spec is a bit unclear on this:

This field MUST be present if the Python implementation has a stable ABI extension suffix, otherwise this entry will be missing.

Does "having a stable ABI extension suffix" imply having a separate suffix for stable ABI extensions? Does it imply having a separate "stable ABI" that's different from regular ABI? Or is it sufficient that PyPy permits building extensions with "limited API", even though PyPy does not define a separate "stable ABI"?

CC @cfbolz, @mattip, @eli-schwartz

[mattip]

PyPy3 (or for that matter, CPython: mattip) can be built either with a shared libpypy3.x-c.so or without

I agree that a Python without a shared *so should not support c-extension modules at all. The cost in complexity for accommodating that very uncommon mode does not justify it. The opposite is true: I have only seen such a build on embedded systems with no dynamic loader that only support vendored c-extensions. I think trying to support dynamic c-extensions in a static build would lead to confusion.

As for stable ABI: yes, it is a separate suffix and would require internal support from PyPy to recognize and load it. There is a long discussion at pypy/pypy#3397.

[eli-schwartz]

Does it imply having a separate "stable ABI" that's different from regular ABI? Or is it sufficient that PyPy permits building extensions with "limited API", even though PyPy does not define a separate "stable ABI"?

If the result of compiling "with the limited API" is the same generated object code as a normal build -- i.e. there is no stable ABI that is different from regular ABI -- then it invites the question, "what is the purpose of having a limited API that doesn't do or mean anything".

Maybe this is a meson bug that it claims to support compiling Limited API extensions with PyPy. Does passing the macro have any effect? It's not entirely uncommon for people working on python support to not have active PyPy experience, and rely on "does it pass the PyPy CI" as a sentinel for "does it make sense to do on PyPy". :(

[eli-schwartz]

PyPy3 (or for that matter, CPython: mattip) can be built either with a shared libpypy3.x-c.so or without

I agree that a Python without a shared *so should not support c-extension modules at all. The cost in complexity for accommodating that very uncommon mode does not justify it. The opposite is true: I have only seen such a build on embedded systems with no dynamic loader that only support vendored c-extensions. I think trying to support dynamic c-extensions in a static build would lead to confusion.

At least on CPython it's no longer the default to link to a shared .so at all, the only point of a .so is for embedding CPython in another application, not extending CPython with a loadable module.

Using shared libraries for the application is totally unrelated to whether the application supports dlopen() / uses ld.so.

[mattip]

Ok, thanks for the correction. On windows CPython no longer builds with a DLL?

[mgorny]

Does passing the macro have any effect?

Not 100% sure if this is meaningful, for at least for psutil (hacked to use limited API on PyPy — normally it does so only on CPython):

$ diffoscope build{,-stable}/lib.linux-x86_64-pypy311/psutil/_psutil_linux.pypy311-pp73-x86_64-linux-gnu.so 
--- build/lib.linux-x86_64-pypy311/psutil/_psutil_linux.pypy311-pp73-x86_64-linux-gnu.so
+++ build-stable/lib.linux-x86_64-pypy311/psutil/_psutil_linux.pypy311-pp73-x86_64-linux-gnu.so
├── objdump --line-numbers --disassemble --demangle --reloc --no-show-raw-insn --section=.text {}
│ @@ -365,15 +365,15 @@
│  
│  0000000000002d1a <PyInit__psutil_linux>:
│  PyInit__psutil_linux():
│  	endbr64
│  	push   %rbp
│  	mov    %rsp,%rbp
│  	sub    $0x10,%rsp
│ -	mov    $0x3f5,%esi
│ +	mov    $0x3,%esi
│  	lea    0x544e(%rip),%rax        
│  	mov    %rax,%rdi
│  	call   26e0 <PyPyModule_Create2@plt>
│  	mov    %rax,-0x8(%rbp)
│  	cmpq   $0x0,-0x8(%rbp)
│  	jne    2d4f <PyInit__psutil_linux+0x35>
│  	mov    $0x0,%eax
│ @@ -1543,15 +1543,15 @@
│  
│  0000000000003e27 <PyInit__psutil_posix>:
│  PyInit__psutil_posix():
│  	endbr64
│  	push   %rbp
│  	mov    %rsp,%rbp
│  	sub    $0x10,%rsp
│ -	mov    $0x3f5,%esi
│ +	mov    $0x3,%esi
│  	lea    0x44c1(%rip),%rax        
│  	mov    %rax,%rdi
│  	call   26e0 <PyPyModule_Create2@plt>
│  	mov    %rax,-0x10(%rbp)
│  	cmpq   $0x0,-0x10(%rbp)
│  	jne    3e5c <PyInit__psutil_posix+0x35>
│  	mov    $0x0,%eax