AMLII-1842 Fix a potential deadlock on fork#836
Merged
Conversation
vickenty
force-pushed
the
vickenty/fork
branch
2 times, most recently
from
af7b9f1 to
7afa4bc
Compare
If a fork happens while some thread is sending metrics and holds one of the locks, the lock would remain locked in the child process and would deadlock, either in the post_fork handler (when the post_fork hook tries to close the socket), or later (when user code tries to send a metric). Work around the issue by resetting the socket and buffer locks in the child process. If those were locked in the parent at the time of the fork, the internal client state may be inconsistent, so we reset it as well. With config lock, we can not reset the state to some known good state, and to avoid problems when fork is called while a thread modifies the client configuration, the config lock will be held across fork. Both the client and the parent can safely unlock it afterwards.
vickenty
added
kind/bug
|
This issue has been automatically marked as stale because it has not had activity in the last 30 days. |
carlosroman
approved these changes
Aug 8, 2024
skarimo
approved these changes
Aug 9, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fix a potential deadlock on fork
Description of the Change
If a fork happens while some thread is sending metrics and holds one of the locks, the lock would remain locked in the child process and would deadlock, either in the post_fork handler (when the post_fork hook tries to close the socket), or later (when user code tries to send a metric).
Work around the issue by resetting the socket and buffer locks in the child process. If those were locked in the parent at the time of the fork, the internal client state may be inconsistent, so we reset it as well.
With config lock, we can not reset the state to some known good state, and to avoid problems when fork is called while a thread modifies the client configuration, the config lock will be held across fork. Both the client and the parent can safely unlock it afterwards.
Alternate Designs
Possible Drawbacks
Verification Process
Additional Notes
Release Notes
Review checklist (to be filled by reviewers)
changelog/label attached. If applicable it should have thebackward-incompatiblelabel attached.do-not-merge/label attached.kind/andseverity/labels attached at least.