RooFit::MultiProcess & TestStatistics part 2b: MultiProcess#8412
Closed
egpbos wants to merge 9 commits into
Closed
RooFit::MultiProcess & TestStatistics part 2b: MultiProcess#8412egpbos wants to merge 9 commits into
egpbos wants to merge 9 commits into
Conversation
The files in here are modified versions of the ZMQ convenience functions from https://gitlab.cern.ch/raaij/generate_and_sort, contributed by @roelaaij.
This is a reimplementation of ZMQ's poll function that mimics the POSIX ppoll function. The difference with regular poll is that ppoll blocks incoming POSIX signals, allowing graceful handling of these signals. We use this in RooFit::MultiProcess to shut down forked child processes. The motivation behind this addition is discussed more in this blog post: https://blog.esciencecenter.nl/combining-zeromq-posix-signals-b754f6f29cd6, or more succinctly in Martin Sustrik's blog post, in the edit at the bottom: https://250bpm.com/blog:12/
The option builtin_zeromq is set to ON by default and controls whether RooFitZMQ is built. At some point, we should also add the option to use a system installed ZeroMQ, so that people can avoid the built-in, if they wish, e.g. for the conda-forge build.
Should have been added in ppoll commit before.
RooFit::MultiProcess is a task-based parallelization framework that will be used to parallelize gradients in RooFit/Minuit2 fits. It uses forked processes for parallelization, as opposed to threads. We chose this approach because A) the existing RooRealMPFE parallelization framework already made use of forks as well, so we had something to build on and B) it was at the time deemed infeasible to check the entire RooFit code for thread-safety. Moreover, we use MultiProcess to parallelize gradients -- i.e. the tasks to be executed in parallel are partial derivatives -- and these are sufficiently large tasks that communication in between tasks is not a big concern in the big fits that we aimed to parallelize. The communication between processes is done using ZeroMQ, accessed through convenience functions in RooFitZMQ. Note: test_Job still fails, it has to be updated to the latest communication framework.
DeepCode's analysis on #fc0ac8 found:
Top issues
👉 View analysis in DeepCode’s Dashboard | Configure the bot👉 The DeepCode service and API will be deprecated in August, 2021. Here is the information how to migrate. Thank you for using DeepCode 🙏 ❤️ !If you are using our plugins, you might be interested in their successors: Snyk's JetBrains plugin and Snyk's VS Code plugin. |
|
Starting build on |
16 tasks
Contributor
Author
|
Made obsolete by #9078. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is the third part of a split and clean-up of #8294, as suggested by @guitargeek.
In this PR, we introduce the RooFit::MultiProcess package. RooFit::MultiProcess is a task-based parallelization framework that will be used to parallelize gradients in RooFit/Minuit2 fits.
It uses forked processes for parallelization, as opposed to threads. We chose this approach because A) the existing RooRealMPFE parallelization framework already made use of forks as well, so we had something to build on and B) it was at the time deemed infeasible to check the entire RooFit code for thread-safety. Moreover, we use MultiProcess to parallelize gradients -- i.e. the tasks to be executed in parallel are partial derivatives -- and these are sufficiently large tasks that communication in between tasks is not a big concern in the big fits that we aimed to parallelize.
The communication between processes is done using ZeroMQ, accessed through convenience functions in RooFitZMQ introduced in #8385.
Notes:
With that in mind, this PR is ready for review.