Error relocating dotnet - missing symbol _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev

[trylek]

Platform: Linux_musl arm64 release
Pipeline: runtime-libraries outerloop, runtime-libraries outerloop-linux
Example run: https://dev.azure.com/dnceng/public/_build/results?buildId=482386&view=logs&j=6c926a84-4f53-5790-1d4c-92b88465ec72&t=bc0eac9e-2cde-5709-c96e-4455fef1ffb5

Proximate diagnostic info:

/root/helix/work/workitem /root/helix/work/workitem
Error relocating /root/helix/work/correlation/dotnet: _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found
/root/helix/work/workitem
----- end Tue Jan 14 11:46:26 UTC 2020 ----- exit code 127 ----------------------------------------------------------
Looking around for any Linux dump...
... found no dump in /root/helix/work/workitem
+ export '_commandExitCode=127'

[ericstj]

Seems like a mismatch between the C runtime we build against vs what we run against?

[janvorli]

We are building Linux MUSL arm64 on Alpine 3.9, this test was running on 3.8. It used to work fine when I was testing it in the past, but maybe we've updated the testing docker images since and a standard C++ library update in 3.9 that we've pulled into the new image has broken the backwards compatibility.
Btw, the mangled name represents std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()
The support for Alpine 3.8 ends on May 1st 2020, so if we officially support it for arm64, it would be good to fix it. However, I am not sure how to do that as the Alpine 3.8 cannot be used for building .NET core - there are some library dependency issues around openssl that were fixed in 3.9, but never back-ported to 3.8.

[janvorli]

There was a problem in building rootfs for Alpine. Specifying the edge/main and edge/testing repos for all the packages lead to an unintended installation of libstdc++.so.6.0.27. The libstdc++.so.6.0.25 from 3.9/main should be installed instead, otherwise the binaries we build have unresolved dependency to std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream() that got apparently added in the latest version of the library.

I've created a PR in arcade to fix that (dotnet/arcade#4657). Once that gets merged and propagated to https://github.com/dotnet/dotnet-buildtools-prereqs-docker, we can regenerate the docker images there and update our lab scripts to use the new image name for building Alpine stuff. That will fix the problem.

[safern]

@janvorli it seems like the way we build rootfs in dotnet-buildtools-prereqs-docker, arcade doesn't need to flow as when we call build-rootfs.sh it clones arcade and uses that fresh clone for the cross scripts: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/master/src/ubuntu/build-scripts/build-rootfs.sh#L35

Since we commented the helix-queues here and the way libraries do testing is not resilient to skipping the send to helix job when the queues are empty, all CI jobs are failing with an error due to the queues being empty.

Can this be fixed already or should I put up a PR to disable the test run for musl arm64?

[janvorli]

I've realized that too. There was a recent change in the https://github.com/dotnet/dotnet-buildtools-prereqs-docker repo (4 days ago), so there should be new images will be generated due to that and we can just use them.
@MichaelSimons do the new images get automatically generated for docker repo PRs or do you need to trigger their build manually?

[MichaelSimons]

@janvorli - the builds automatically run when a PR is merged. The entire set of images do not get rebuilt, only the Dockerfiles affected by the PR are built.

Unfortunately there was an infrastructure issue that impacted updating the versions repo with the latest images. If you tell me which image you are looking for I can look up the latest version and let you know.

[janvorli]

@MichaelSimons the image we need to get is the new version of mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-arm64-alpine-406629a-20191023143847. We need the rootfs in it to be rebuilt based on a recent change in the Arcade repo. As there was no change in the related Dockerfile, based on what you've said above I am not sure if it got built.

[MichaelSimons]

@janvorli - the most recent image is ubuntu-16.04-cross-arm64-alpine-406629a-20200121150126. It was rebuilt as a result of merging this change. If you need a newer build, let me know and I can trigger the build.

[janvorli]

@MichaelSimons I've checked the image you've mentioned and it doesn't have the necessary change in the rootfs. Could you please trigger a build of a newer one?

[MichaelSimons]

The build completed. The newest tag is now ubuntu-16.04-cross-arm64-alpine-406629a-20200127195039