compilation fails on FreeBSD 13.2 RELEASE

[mkrzewic]

• [x ] Checked for duplicates

Describe the bug

segfault in build process.

Expected behavior

build process completes.

To Reproduce

standard cmake build process, as described in the guide.

Setup

ROOT v6-26-10 (@4dddea35)
FreeBSD 13.2 RELEASE (clang 14.0.5), required dependencies installed from ports

Additional context

a port exists v6-28-02 (development/root), installs in binary form (so somebody got it to compile, did not have too much time to test it thoroughly). In my case, however, the port also does not compile, craps out somewhere else. plus it hauls in too many dependencies so it is less than ideal.
I imagine getting it to work on FreeBSD would not end at fixing this issue here. The fbsd port maintainer has some patches which might help, but I have also seen that e.g. procfs is deprecated in new releases and libprocstat has to be used, etc, etc.
With that being said, it would be extremely nice to get ROOT to work on FreeBSD...
thanks,

[ 52%] Built target rootcling_stage1
[ 52%] Generating G__Core.cxx, ../lib/Core.pcm
 #0 0x000000000446e3d5 llvm::sys::PrintStackTrace(llvm::raw_ostream&) Signals.cpp:0:0
 #1 0x000000000446c646 llvm::sys::RunSignalHandlers() Signals.cpp:0:0
 #2 0x000000000446eba6 SignalHandler(int) Signals.cpp:0:0
 #3 0x0000000826961b60 (/lib/libthr.so.3+0x19b60)
 #4 0x000000082696111f (/lib/libthr.so.3+0x1911f)
 #5 0x00007ffffffff8a3 ([vdso]+0x2d3)
 #6 0x000000082da5f7b4 strlen (/lib/libc.so.7+0x1587b4)
 #7 0x00000000016482c9 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string<std::nullptr_t>(char const*) rootcling_stage1.cxx:0:0
 #8 0x0000000001648213 GetEtcDir() rootcling_stage1.cxx:0:0
 #9 0x00000000016b2792 RootClingMain(int, char**, bool) (/home/mkrzewic/src/root_build/core/rootcling_stage1/src/rootcling_stage1+0x16b2792)
#10 0x00000000016c32d5 ROOT_rootcling_Driver (/home/mkrzewic/src/root_build/core/rootcling_stage1/src/rootcling_stage1+0x16c32d5)
Stack dump:
0.	Program arguments: /home/mkrzewic/src/root_build/core/rootcling_stage1/src/rootcling_stage1 -v2 -f G__Core.cxx -cxxmodule -s /home/mkrzewic/src/root_build/lib/libCore.so -excludePath /home/mkrzewic/src/root_src -excludePath /home/mkrzewic/src/root_build/ginclude -excludePath /home/mkrzewic/src/root_build/externals -excludePath /home/mkrzewic/src/root_build/builtins -writeEmptyRootPCM -m _Builtin_intrinsics -mByproduct _Builtin_intrinsics -mByproduct ROOT_Foundation_Stage1_NoRTTI -mByproduct ROOT_Foundation_C -mByproduct ROOT_Rtypes -D__STDC_NO_COMPLEX__ -D__COMPLEX_H__ -D_COMPLEX_H -D__CLANG_STDATOMIC_H -compilerI/usr/include/c++/v1 -compilerI/usr/lib/clang/14.0.5/include -compilerI/usr/include -compilerI/usr/lib/clang/14.0.5/include -compilerI/usr/include -I/home/mkrzewic/src/root_build/include -I/home/mkrzewic/src/root_build/ginclude -I/home/mkrzewic/src/root_src/core/base/inc -I/home/mkrzewic/src/root_src/core/foundation/inc -I/home/mkrzewic/src/root_src/core/cont/inc -I/home/mkrzewic/src/root_src/core/gui/inc -I/home/mkrzewic/src/root_src/core/meta/inc -I/home/mkrzewic/src/root_src/core/clib/inc -I/home/mkrzewic/src/root_src/core/rint/inc -I/home/mkrzewic/src/root_src/core/zip/inc -I/home/mkrzewic/src/root_src/core/thread/inc -I/home/mkrzewic/src/root_src/core/textinput/inc -I/home/mkrzewic/src/root_src/core/clingutils/inc -I/home/mkrzewic/src/root_src/core/base/v7/inc -I/home/mkrzewic/src/root_src/core/foundation/v7/inc -I/home/mkrzewic/src/root_src/core/unix/inc -I/usr/local/include -I/home/mkrzewic/src/root_src/builtins/xxhash -I/usr/include ROOT/TErrorDefaultHandler.hxx ROOT/TSequentialExecutor.hxx ROOT/StringConv.hxx Buttons.h Bytes.h Byteswap.h KeySymbols.h MessageTypes.h Riostream.h Rtypes.h TApplication.h TAtt3D.h TAttAxis.h TAttBBox2D.h TAttBBox.h TAttFill.h TAttLine.h TAttMarker.h TAttPad.h TAttText.h TBase64.h TBenchmark.h TBuffer3D.h TBuffer3DTypes.h TBuffer.h TColor.h TColorGradient.h TDatime.h TDirectory.h TEnv.h TException.h TExec.h TFileCollection.h TFileInfo.h TFolder.h TInetAddress.h TMacro.h TMathBase.h TMD5.h TMemberInspector.h TMessageHandler.h TNamed.h TNotifyLink.h TObject.h TObjString.h TParameter.h TPluginManager.h TPoint.h TPRegexp.h TProcessID.h TProcessUUID.h TQClass.h TQCommand.h TQConnection.h TQObject.h TRedirectOutputGuard.h TRefCnt.h TRef.h TRegexp.h TRemoteObject.h TROOT.h TRootIOCtor.h TStopwatch.h TStorage.h TString.h TStringLong.h TStyle.h TSysEvtHandler.h TSystemDirectory.h TSystemFile.h TSystem.h TTask.h TThreadSlots.h TTime.h TTimer.h TTimeStamp.h TUri.h TUrl.h TUUID.h TVersionCheck.h TVirtualAuth.h TVirtualFFT.h TVirtualGL.h TVirtualMonitoring.h TVirtualMutex.h TVirtualPadEditor.h TVirtualPad.h TVirtualPadPainter.h TVirtualPerfStats.h TVirtualPS.h TVirtualQConnection.h TVirtualRWMutex.h TVirtualTableInterface.h TVirtualViewer3D.h TVirtualX.h ROOT/RDirectoryEntry.hxx ROOT/RIndexIter.hxx strlcpy.h snprintf.h strtok.h ROOT/TSeq.hxx TArrayC.h TArrayD.h TArrayF.h TArray.h TArrayI.h TArrayL64.h TArrayL.h TArrayS.h TBits.h TBtree.h TClassTable.h TClonesArray.h TCollection.h TCollectionProxyInfo.h TExMap.h THashList.h THashTable.h TIterator.h TList.h TMap.h TObjArray.h TObjectTable.h TOrdCollection.h TRefArray.h TRefTable.h TSeqCollection.h TSortedList.h TVirtualCollectionProxy.h ESTLType.h RStringView.h TClassEdit.h TError.h ROOT/RLogger.hxx ROOT/RNotFn.hxx ROOT/RRangeCast.hxx ROOT/RSpan.hxx ROOT/RStringView.hxx ROOT/StringUtils.hxx ROOT/span.hxx ROOT/TypeTraits.hxx TUnixSystem.h root_std_complex.h GuiTypes.h TApplicationImp.h TBrowser.h TBrowserImp.h TCanvasImp.h TClassMenuItem.h TContextMenu.h TContextMenuImp.h TControlBarImp.h TGuiFactory.h TInspectorImp.h TObjectSpy.h TToggleGroup.h TToggle.h TBaseClass.h TClassGenerator.h TClass.h TClassRef.h TClassStreamer.h TDataMember.h TDataType.h TDictAttributeMap.h TDictionary.h TEnumConstant.h TEnum.h TFileMergeInfo.h TFunction.h TFunctionTemplate.h TGenericClassInfo.h TGlobal.h TInterpreter.h TInterpreterValue.h TIsAProxy.h TListOfDataMembers.h TListOfEnums.h TListOfEnumsWithLock.h TListOfFunctions.h TListOfFunctionTemplates.h TMemberStreamer.h TMethodArg.h TMethodCall.h TMethod.h TProtoClass.h TRealData.h TSchemaHelper.h TSchemaRule.h TSchemaRuleSet.h TStatusBitsChecker.h TStreamerElement.h TStreamer.h TVirtualIsAProxy.h TVirtualRefProxy.h TVirtualStreamerInfo.h TVirtualArray.h TVirtualObject.h Getline.h /home/mkrzewic/src/root_src/core/base/inc/LinkDef.h 
Segmentation fault
gmake[2]: *** [core/CMakeFiles/G__Core.dir/build.make:474: core/G__Core.cxx] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:32925: core/CMakeFiles/G__Core.dir/all] Error 2
gmake: *** [Makefile:156: all] Error 2

[hahnjo]

Hi @mkrzewic, indeed ROOT currently does not compile on FreeBSD. The crash you are seeing is because ROOTSYS is not set, which is a (environment) variable that should be set automatically as needed. The port has a patch for this: https://github.com/freebsd/freebsd-ports/blob/cfa050b3a4535cce59341631524e758ce6fc0ba2/devel/root/files/patch-core_dictgen_src_rootcling__impl.cxx However, applying it doesn't actually work on FreeBSD 13.2 because the method enabled by the patch relies on /proc which is empty - as you say, this needs to migrate to a different method (in this case for finding the path of the current executable).

Unfortunately, I fear that there are currently no resources available to support FreeBSD and then maintain it. If you want to give it a shot yourself, essentially cleaning up the available patches and properly submitting them, please feel free to ping me on the pull requests 😃

[mkrzewic]

Hi @hahnjo , I found some time to look into it and have a question since the build system around the dict generation is a bit complicated: where would be the place to add library dependencies for rootcling_stage1 ?
specifically the problem is:
`
[ 78%] Linking CXX executable src/rootcling_stage1
[ 78%] Built target Cling
ld: error: undefined symbol: procstat_open_sysctl

referenced by rootcling_impl.cxx
../dictgen/CMakeFiles/Dictgen.dir/src/rootcling_impl.cxx.o:(GetExePath())

ld: error: undefined symbol: kinfo_getproc

referenced by rootcling_impl.cxx
../dictgen/CMakeFiles/Dictgen.dir/src/rootcling_impl.cxx.o:(GetExePath())

ld: error: undefined symbol: procstat_getpathname

referenced by rootcling_impl.cxx
../dictgen/CMakeFiles/Dictgen.dir/src/rootcling_impl.cxx.o:(GetExePath())
c++: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [core/rootcling_stage1/CMakeFiles/rootcling_stage1.dir/build.make:319: core/rootcling_stage1/src/rootcling_stage1] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:43752: core/rootcling_stage1/CMakeFiles/rootcling_stage1.dir/all] Error 2
gmake: *** [Makefile:156: all] Error 2
`

[hahnjo]

@mkrzewic

where would be the place to add library dependencies for rootcling_stage1 ?

That would be core/rootcling_stage1/CMakeLists.txt, I think. ROOT_EXECUTABLE is one of our macros, it will generate a target that you can call the usual CMake functions on.

[mkrzewic]

I'm getting a crash at the end, while I investigate maybe some of you recognize it:
[100%] Generating lib/modules.idx cling::DynamicLibraryManager::loadLibrary(): /home/mkrzewic/build/root/bin/root.exe: mmap of entire address space failed: Cannot allocate memory

stacktrace attached.
root_crash.log

[hahnjo]

(assuming you haven't found it yet)

Hm, looking at the code in DynamicLibraryManager::loadLibrary() this seems to (eventually) come from a call to the (platform) dlopen and dlerror. Since it mentions memory allocation and mmap in particular, are you maybe running out of memory or are there ulimits that get into the way? Otherwise can you maybe push your (WIP) changes somewhere so I could have a look? Maybe I can spot something when I can run it myself...

[mkrzewic]

@hahnjo hi, my WIP branch is https://github.com/mkrzewic/root/tree/FreeBSD
I didn't have alot of time to look yet, have a completely new issue after rebuilding with Debug on... :)
I don't think (I hope not) it's a memory issue, no ulimits, 8GB on the machine (plus plenty of swap). It is in a jail though, but also there no limits. I'll move to a bigger machine later today if I find time.

[mkrzewic]

@hahnjo Hi, I finally found some time to get this building, the branch now seems to somewhat work. Few things:

• it seems not to break the linux build
• seems to work for now, in whatever simple scenarios I have tested it till now
• the TThread tests crap out saying the _REENTRANT macro is not defined (I think it should be by -pthread which is enabled), probably needs some attention by an expert.
• there is an issue with library unload order vs the thread local dtor calling. Not a dealbreaker but the FreeBSD __cxa_thread_call_dtors is a bit verbose and prints warnings to stderr about that. Also here there is quite some room for experts to look at since there already are mitigations to similar issues in the code. (__cxa_thread_call_dtors: dtr 0x82b2c78f0 from unloaded dso, skipping)
• actually a lot more tests fail - what is the general state of the test results on the master branch?

[hahnjo]

@mkrzewic great work! I haven't looked at the changes in detail yet, just at the commits from a high level; for the first one in particular, it would be great to split this into smaller logical pieces. For example, the linking of the crypt library looks fairly self-contained. I'm not sure if it's a good idea to disguise FreeBSD as Linux by including SetUpLinux, did you evaluate creating a separate SetUpBSD or something like that?

the TThread tests crap out saying the _REENTRANT macro is not defined (I think it should be by -pthread which is enabled), probably needs some attention by an expert.

Looking at interpreter/llvm/src/tools/clang/lib/Basic/Targets/OSTargets.h, it indeed seems to not define _REENTRANT. I agree that it needs a separate look, probably not critical for the first support.

actually a lot more tests fail - what is the general state of the test results on the master branch?

Tests should pass on Linux. It's possible that there is some common problem that makes multiple / many tests fail. Probably needs to be looked at in more detail.

[mkrzewic]

@hahnjo

I'm not sure if it's a good idea to disguise FreeBSD as Linux by including SetUpLinux, did you evaluate creating a separate SetUpBSD or something like that?

indeed I have thought about it, but laziness took over at this stage since fbsd is apparently similar enough for a first try. I'll separate it out.

Tests should pass on Linux. It's possible that there is some common problem that makes multiple / many tests fail. Probably needs to be looked at in more detail.

luckily most tests seem to pass on linux with my branch. The failed ones I think are related to whatever I did not want to install (dependency-wise)
On FreeBSD the situation might be very similar since I keep a pretty minimal system. A couple do have me worried though, like the threading stuff and there is a number of std container/wrapper streaming tests that fail (e.g. roottest-root-io-cpp11Containers-forwardList) plus some more IO related stuff, I suspect some different conventions in the linking chain, who knows, a root internals expert might find the issue much faster than I can.

[mkrzewic]

@hahnjo I reorganized the patches for a more fine grained overview. I did leave one larger patch (the initial one) as a throw-in-all bucket to have a good placeholder for creds to the port maintainers

[hahnjo]

I reorganized the patches for a more fine grained overview. I did leave one larger patch (the initial one) as a throw-in-all bucket to have a good placeholder for creds to the port maintainers

Excellent, thanks @mkrzewic! Can you open a PR for a proper review? The structure looks good to me, just some minor details to clarify and other people to have a look.