macOS CI failing

[dstansby]

macOS tests have started failing, with no obvious culprit. The last passing test run is here, and an example failing run here.

The error is in test_vlen_bytes.py::test_encode_none :

 Fatal Python error: Aborted

Current thread 0x00007ff85e49a9c0 (most recent call first):
  File "/Users/runner/work/numcodecs/numcodecs/numcodecs/tests/test_vlen_bytes.py", line 90 in test_encode_none
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/_pytest/python.py", line 159 in pytest_pyfunc_call
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/_pytest/python.py", line 1627 in runtest
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/_pytest/runner.py", line 174 in pytest_runtest_call
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/_pytest/runner.py", line 242 in <lambda>
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/_pytest/runner.py", line 241 in call_and_report
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/_pytest/runner.py", line 132 in runtestprotocol
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/_pytest/main.py", line 362 in pytest_runtestloop
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/_pytest/main.py", line 337 in _main
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/_pytest/main.py", line 283 in wrap_session
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/Users/runner/miniconda3/envs/test/lib/python3.11/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/Users/runner/miniconda3/envs/test/bin/pytest", line 8 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, numcodecs.compat_ext, numcodecs.blosc, numcodecs.zstd, numcodecs.lz4, numcodecs._shuffle, numcodecs.jenkins, crc32c._crc32c, numcodecs.vlen, numcodecs.fletcher32, msgpack._cmsgpack, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator (total: 21)
/Users/runner/work/_temp/7d2eafc4-94de-4bcd-acc8-5e21a4e9db3a.sh: line 1: 12992 Abort trap: 6           pytest -v
numcodecs/tests/test_vlen_bytes.py::test_encode_none 

[dstansby]

The only differences in the coda environment are
conda version : 24.11.0 / conda version : 24.11.2

[jakirkham]

When running the test VLenBytes using a round-trip of encode/decode with the data from the test

numcodecs/numcodecs/tests/test_vlen_bytes.py

Line 92 in 58bc9c4

a = np.array([b'foo', None, b'bar'], dtype=object)

Can see the following error under lldb

* thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert
    frame #4: 0x0000000118bf4c58 vlen.cpython-312-darwin.so`__pyx_pw_9numcodecs_4vlen_9VLenBytes_3encode + 2948
vlen.cpython-312-darwin.so`__pyx_pw_9numcodecs_4vlen_9VLenBytes_3encode:
->  0x118bf4c58 <+2948>: adr    x0, 0x118bfe6b6 ; "PyBytes_GET_SIZE"

This appears to come from this Numcodecs code. Specifically the call to PyBytes_GET_SIZE at the end

numcodecs/numcodecs/vlen.pyx

Lines 248 to 253 in 58bc9c4

	b = values[i]
	if b is None or b == 0: # treat these as missing value, normalize
	b = b''
	elif not PyBytes_Check(b):
	raise TypeError('expected byte string, found %r' % b)
	l = PyBytes_GET_SIZE(b)

Within PyBytes_GET_SIZE, it performs a cast that first does assert. It is that assert, which fails and causes the segfault

Though looking at the values and our code, they appear to be bytes values or are converted to bytes values. Also PyBytes_Check should catch anything that is not bytes and error. For some reason this does not appear to work