Migrate pybind11 to nanobind

[justinchuby]

This pull request makes several significant changes to the Python extension build and C++/Python interop layers. The most important update is the migration from pybind11 to nanobind for Python bindings, which affects both the C++ code and the build system. It also updates the supported Python versions for builds and wheels, and introduces improved handling for Python limited API wheels.

Migration from pybind11 to nanobind:

• Replaces all usage of pybind11 with nanobind in C++ source files such as pyop/py_c_api.cc and pyop/pyfunc.cc, updating type aliases, includes, and APIs accordingly. This includes changes to type conversions, error handling, and NumPy array creation. [1] [2] [3] [4] [5] [6] [7] [8]
• Updates the CMake build scripts to fetch and use nanobind instead of pybind11, including removing the pybind11 external and adding a new nanobind.cmake module. The extension module is now built with nanobind_add_module and linked accordingly. [1] [2] [3] [4]

Build and wheel configuration updates:

• Drops support for building and testing with Python 3.7, 3.8, and 3.9 across all CI/CD workflows and build scripts, now supporting only Python 3.10, 3.11, and 3.12. This affects Linux, Windows, and macOS builds. [1] [2] [3] [4] [5] [6]

Wheel build improvements:

• Adds a custom bdist_wheel command to ensure that wheels built with Python 3.12 and above are marked as limited API (cp312), improving compatibility. [1] [2]

CMake and Python discovery improvements:

• Refactors Python discovery in cmake/ext_python.cmake to use the modern Python package instead of Python3, improving reliability and future-proofing the build process.

Dependency tracking:

• Updates the dependency manifest to point to the nanobind repository and its current version, replacing the previous reference to pybind11.

[Copilot]

Pull request overview

Migrates the project’s Python extension bindings and build integration from pybind11 to nanobind, while also tightening the supported Python versions and improving limited-API (abi3) wheel handling across build/CI.

Changes:

• Replaced pybind11-based bindings with nanobind in the Python custom-op layer (pyop/*) and updated module initialization.
• Updated CMake to fetch/build with nanobind and modernized Python discovery via find_package(Python ...).
• Dropped older Python versions from CI wheel matrices and added limited-API wheel tagging plus abi3 verification in wheel test scripts.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tools/test_cibuildwheel.sh	Adds import smoke-test and runs abi3audit for Python ≥ 3.12.
tools/test_cibuildwheel.bat	Windows equivalent of the wheel import check + abi3audit.
tools/ci_build/github/azure-pipeline/wheels_macos.yml	Removes cp313 from wheel build matrix.
tools/ci_build/github/azure-pipeline/wheels_linux.yml	Removes cp313 from wheel build matrix.
tools/ci_build/github/azure-pipeline/templates/windows-build-stage.yml	Removes cp313 from wheel build matrix.
setup.py	Adds python_requires>=3.10 and marks the extension as limited-API when applicable.
pyop/pykernel.h	Updates binding API surface to accept nanobind::module_.
pyop/pyfunc.cc	Migrates bindings and NumPy interop from pybind11 to nanobind.
pyop/py_c_api.cc	Migrates C-API bindings and NumPy interop from pybind11 to nanobind.
cmake/externals/pybind11.cmake	Removes pybind11 FetchContent external.
cmake/externals/nanobind.cmake	Adds nanobind FetchContent external.
cmake/ext_python.cmake	Switches Python discovery to Python package and builds the extension with nanobind_add_module.
cgmanifest.json	Updates dependency tracking from pybind11 to nanobind.
.pyproject/cmdclass.py	Adds a custom bdist_wheel to mark CPython ≥ 3.12 wheels as limited-API.
.pipelines/wheels_win32.yml	Drops older Python versions from wheel build matrix.
.pipelines/wheels_macos.yml	Drops older Python versions from wheel build matrix.
.pipelines/wheels_linux.yml	Drops older Python versions from wheel build matrix.
.pipelines/templates/onebranch-windows-build-stage.yml	Drops older Python versions from wheel build matrix.
.github/workflows/linux_arm64_wheel.yaml	Updates wheel build workflow to Python 3.10–3.12 only.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[justinchuby]

E       onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to get symbol RegisterCustomOps with error: /mnt/azure_nvme_temp/_work/1/s/onnxruntime_extensions/_extensions_pydll.abi3.so: undefined symbol: RegisterCustomOps

test_string_ops.py:978: Fail
_________________ TestPythonOpString.test_string_split_cc_sep0 _________________

self = <test_string_ops.TestPythonOpString testMethod=test_string_split_cc_sep0>

    def test_string_split_cc_sep0(self):
        so = _ort.SessionOptions()
>       so.register_custom_ops_library(_get_library_path())
E       onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to get symbol RegisterCustomOps with error: /mnt/azure_nvme_temp/_work/1/s/onnxruntime_extensions/_extensions_pydll.abi3.so: undefined symbol: RegisterCustomOps

test_string_ops.py:1079: Fail

[justinchuby]

Merging into a branch in this repo first to facilitate copilot edits