Stop passing URL opener across process boundaries

[nanonyme]

Should fix #1766

The object is not picklable. This cannot be supported. Instead create URL opener when needed in child process. Drops URL opener caching which may result in minor performance impact.

[nanonyme]

Looks like this breaks netrfc functionality. Need to debug.

[nanonyme]

I suspect this is CI-only problem. Environment is working in unexpected way.

[nanonyme]

Okay, done. I think there is some problem with buildgrid that is causing test failures.

[BenjaminSchubert]

I like the change, but I am a bit cautious about the invalid netrc error, I am not sure I get why this can't be logged anymore?

[nanonyme]

Failing test appears to have been failed at [00:00:00] FAILURE [4a47c98a] build-shell/buildtree.bst: Failed to link '/home/testuser/buildstream/.tox/py310/tmp/test_shell_pull_cached_buildtr2/cache/artifacts/refs/test/build-shell-buildtree/4a47c98a10df39e65e99d471f96edc5b58d4ea5b9b1f221d0be832a8124b8099 -> /home/testuser/buildstream/.tox/py310/tmp/test_shell_pull_cached_buildtr2/cache/artifacts/refs/test/build-shell-buildtree/38b730aec720dbddba843c9e3ce1ad8073e39d16cc4d25f752bc89d867fbaec1': [Errno 17] File exists: '/home/testuser/buildstream/.tox/py310/tmp/test_shell_pull_cached_buildtr2/cache/artifacts/refs/test/build-shell-buildtree/4a47c98a10df39e65e99d471f96edc5b58d4ea5b9b1f221d0be832a8124b8099' -> '/home/testuser/buildstream/.tox/py310/tmp/test_shell_pull_cached_buildtr2/cache/artifacts/refs/test/build-shell-buildtree/38b730aec720dbddba843c9e3ce1ad8073e39d16cc4d25f752bc89d867fbaec1'

[abderrahim]

Failing test appears to have been failed at [...]

Which seems to be the same as #1814. I guess that's just a flaky test.

[nanonyme]

It looks strange. Apparently the error originates here https://github.com/apache/buildstream/blob/master/src/buildstream/utils.py#L432 and it looks like bst should upon file existing try to delete it and then try to link again.

[nanonyme]

It definitely looks like a race condition with multiple parties trying create same link. Supposedly a chain

1. A creates link
2. B tries to create link and gets error it exists
3. C tries to create link and gets error it exists
4. B deletes link
5. C tries to delete link and silently fails
6. B creates link
7. C tries to create link and throws exception

can result in this failure. safe_link is not multi-thread or multi-process safe.

[nanonyme]

The should probably be a follow-up issue on that. I suspect there might be something specific to test runner that results in this since I haven't ever seen this in real world.

[gtristan]

Rebased this and fixed conflicts with downloadablefilesource.py, this PR is now replaced by #1826