Add support for Python 3.12

[nanonyme]

Continuation of #1872

[abderrahim]

So I tried to run the tests locally, and it seems we can't easily get 3.7 and 3.12 at the same time. 3.12 doesn't work with the current dependencies and trying to update them seems to break 3.7.

[juergbi]

So I tried to run the tests locally, and it seems we can't easily get 3.7 and 3.12 at the same time. 3.12 doesn't work with the current dependencies and trying to update them seems to break 3.7.

I suppose that's not surprising given that Python 3.7 reached end of life 5 months ago. I think it's time to drop Python 3.7 support in BuildStream master as well.

[nanonyme]

@juergbi who needs to be present for that decision?

[nanonyme]

@abderrahim both pyroaring and ruamel.yaml.clib claim to support Python 3.7 with the new versions that support Python 3.12. Was the problematic dependency grpcio?

[nanonyme]

@juergbi can I have another pipeline? It would be useful to be able to run pipelines without approval every time.

[nanonyme]

Looks like grpcio 1.59.0 or newer is required for Python 3.12 support. These should support Python 3.12.

[nanonyme]

I am seeing if I can surgically create dependencies that work on all Python versions. It would be better to drop Python 3.7 support in a separate PR if possible.

[nanonyme]

@juergbi there's some random errors with buildbox-casd. It's problematic that tests do not propagate buildbox-casd logs when they fail.

[abderrahim]

@juergbi there's some random errors with buildbox-casd. It's problematic that tests do not propagate buildbox-casd logs when they fail.

If this is the buildgrid service-tests, you can see that buildgrid is crashing because it's failing to import redis

[nanonyme]

@abderrahim no, I mean Debian 10, Fedora 37 and Fedora 38. There's comments about buildbox-casd not starting as expected. And we're also seeing the pesky hardlink failures.

    [00:00:00] FAILURE [4a47c98a] build-shell/buildtree.bst: Failed to link '/home/testuser/buildstream/.tox/py311/tmp/test_shell_pull_cached_buildtr2/cache/artifacts/refs/test/build-shell-buildtree/4a47c98a10df39e65e99d471f96edc5b58d4ea5b9b1f221d0be832a8124b8099 -> /home/testuser/buildstream/.tox/py311/tmp/test_shell_pull_cached_buildtr2/cache/artifacts/refs/test/build-shell-buildtree/38b730aec720dbddba843c9e3ce1ad8073e39d16cc4d25f752bc89d867fbaec1': [Errno 17] File exists: '/home/testuser/buildstream/.tox/py311/tmp/test_shell_pull_cached_buildtr2/cache/artifacts/refs/test/build-shell-buildtree/4a47c98a10df39e65e99d471f96edc5b58d4ea5b9b1f221d0be832a8124b8099' -> '/home/testuser/buildstream/.tox/py311/tmp/test_shell_pull_cached_buildtr2/cache/artifacts/refs/test/build-shell-buildtree/38b730aec720dbddba843c9e3ce1ad8073e39d16cc4d25f752bc89d867fbaec1'

[nanonyme]

Beyond these it seems tests now pass.

[nanonyme]

Can I have another pipeline? I pushed commit that may stabilize tests.

[nanonyme]

The commit did not help.
tests/integration/shellbuildtrees.py::test_shell_pull_cached_buildtree[pull-without-buildtree] FAILED [ 72%] this is still esoterically failing due to file existing that should not.

[nanonyme]

It looks like it's always the same test that fails.

[abderrahim]

It looks like it's always the same test that fails.

Yeah. It looks like some race condition or something. We need to figure out where it is coming from :/

[abderrahim]

It looks like it's always the same test that fails.

Yeah. It looks like some race condition or something. We need to figure out where it is coming from :/

Indeed it is. For some reason, bst is launching 2 jobs at once to pull the same element.

[abderrahim]

@nanonyme #1882 fixes the "pesky hardlink failure", can you please rebase now that it's merged.

[nanonyme]

@abderrahim pushed, removed the last commit that was irrelevant.

[nanonyme]

Only buildgrid test is now failing.

[abderrahim]

Yup, and I believe it's the buildgrid image that's broken (and our tests broke because we're using the nightly image of buildgrid rather than a fixed version)