[4.0] rabbitmq: block client port on startup#1563
Merged
vuntz merged 1 commit intoMay 3, 2018
Merged
Conversation
ilausuch
force-pushed
the
rabbitmq_block_client_port_on_startup_backport
branch
from
deea4a0 to
f60a1cf
Compare
ilausuch
changed the title
ilausuch
force-pushed
the
rabbitmq_block_client_port_on_startup_backport
branch
3 times, most recently
from
edcc318 to
9330768
Compare
Contributor
Author
1 similar comment
Contributor
Author
ilausuch
changed the title
ilausuch
force-pushed
the
rabbitmq_block_client_port_on_startup_backport
branch
2 times, most recently
from
a4b3f22 to
9f9b50d
Compare
vuntz
suggested changes
Mar 9, 2018
vuntz
left a comment
vuntz
left a comment
Member
There was a problem hiding this comment.
See comments in PR against master
ilausuch
force-pushed
the
rabbitmq_block_client_port_on_startup_backport
branch
from
9f9b50d to
8e13cf0
Compare
ilausuch
force-pushed
the
rabbitmq_block_client_port_on_startup_backport
branch
from
8e13cf0 to
296999d
Compare
This script blocks the connection to the rabbitmq cluster in case the number of nodes decay bellow the half of the total. In this case the remain rabbit nodes won't accept new connections if the quorum is reached. It take advantage of the pacemaker notifications that notify when a rabbitmq has failed or has restored. All nodes rabbitmq ports will be blocked if the total number of alive nodes are below to the half of nodes of the cluster, or unblock if its over this value. The script is divided in two parts: On one hand, the alert handler that manages the alerts discarding non interesting alerts, launches to the blocker script in this node and other cluster nodes (via SSH). And in the other hand, the blocker script than checks the condition of running nodes and blocks or unblocks the rabbitmq client port. (cherry picked from commit 60fb8ce)
ilausuch
force-pushed
the
rabbitmq_block_client_port_on_startup_backport
branch
from
296999d to
8342a7d
Compare
vuntz
approved these changes
Apr 13, 2018
Itxaka
approved these changes
Apr 13, 2018
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This script blocks the connection to the rabbitmq cluster in case the number of nodes decay bellow the half of the total. In this case the remain rabbit nodes won't accept new connections. The problem to solve is that when a rabbit node has too much clients with low timeout, the final scenario could be a DOS when the clients try to constantly reconnect to rabbitmq and forcing more timeouts.
It takes advantage of the pacemaker notifications that notify when a rabbitmq has failed or has restored. All nodes rabbitmq ports will be blocked if the total number of alive nodes are below to the half of nodes of the cluster, or unblock if its over this value. This prevents the overload of alive nodes and the total block of service.
Only the node that have problems will receive the alert, so it is necessary to notify other nodes to block their ports if necessary. The only case when this script won't work is when the node fails completely and is not able to notify the others.
The script is divided in two parts: On one hand, the alert handler that manages the alerts discarding non interesting alerts, launches to the blocker script in this node and other cluster nodes (via SSH). And in the other hand, the blocker script than checks the condition of running nodes and blocks or unblocks the rabbitmq client port.
(cherry picked from commit 60fb8ce)
Backport from #1554