Fix: sbd-cluster: periodically check connection to corosync-daemon - #76
Fix: sbd-cluster: periodically check connection to corosync-daemon#76wenningerk wants to merge 1 commit into
Conversation
1c2cb15 to
faad12d
Compare
|
I think that here is more suitable place for discussion about this PR. :)
Any healthy check are not necessary, because all must work reasonably as is. :) But sometimes healthy checks checking some trivial. But no matter, I'll happy and without node id check. But what about watchdog the healthy node. I have:
When I do
And the same on the second:
configs (-v -I 30 is wroten and commented by me, the default is without -v -I 30):
|
faad12d to
d7a3c67
Compare
|
Looks as if you are RT budgeting in your setup ('sched: RT throttling activated'). |
Say me how to check this. My setup is a default, I didn't change something so smart. I can only add that my test bed (correct? 🧐, the apple dictionary suggest such word) is on the VirtualBox under MacOs.
As I said I'll be happy with just to add '-I 30' to my setup. The only reason why I mention all this, is that the HA system must work as expected with a default setup of a common user. To not to fix the same place twice. If you think that this problem is the only my special case, fine by me. I am looking forward for your patch in git. |
|
Do you get logs like "Unable to set scheduler priority to ..." from sbd. |
| #endif | ||
| /* TODO - Make a CPG call and only call notify_parent() when we get a reply */ | ||
| notify_parent(); | ||
| break; |
There was a problem hiding this comment.
This will be unreachable code if SUPPORT_COROSYNC && CHECK_CPG_HANDLE is true but HAVE_DECL_PCMK_CLUSTER_CMAN is false -- I'd just separate the corosync and cman blocks
| so going ahead with whatever found in | ||
| cluster.cpg_handle should be safe | ||
| */ | ||
| if (cpg_local_get(cluster.cpg_handle, &local_nodeid) == CS_OK) { |
There was a problem hiding this comment.
CPG doesn't need to be connected (cluster_connect_cpg()) before calling cpg_local_get()?
cluster will be initialized to all 0's because it's static, so I don't think there will be any randomness. Which is good because otherwise static analysis might complain about using uninitialized memory :)
There was a problem hiding this comment.
It needs the connection but that code is on pacemaker-side in crm_cluster_connect.
Question is if it wouldn't be cleaner to have something like crm_cluster_check in pacemaker doing basically cpg_local_get or whatever seems to be useful for the cluster-type.
Initially there is gonna be a 0 but 0 is as well a valid cpg_handle. So checking for 0 isn't safe.
Haven't run coverity over it but using static-memory shouldn't result in complaints I guess.
The randomness comes from corosync which adds a random number to the cpg_handle.
I stopped corosync on one node ( Second, |
|
Ok that looks if CPUAccounting isn't enabled. Good so far for rt-scheduling ;-) |
Don't sure that the problem is in RT-scheduling. I see the time problem is roughly 20 seconds, don't sure that this is due to scheduling.
From
As I said this is a "test bed" (a correct word? :)) MacBook Pro with installed VirtualBox. Testing VMs, that forms a Pacemaker cluster, are under VirtualBox. |
|
If your test environment is VMs in VirtualBox on a MacBook Pro I guess you have to expect certain timings needed to be relaxed. |
Don't think so. Every VM consume only one virtual CPU kernel, while CPU has 12 physical kernels.
Just power off (hardly, from VirtualBox manager) one of the VM. The second VM was watchdoged. |
Well, maybe tight on some other resource and blocking scheduling of the virtual CPU ... as said desktop-hypervisors are not necessarily made to separate virtual workloads as the ones made for running loads in datacenters (KVM, ESX, XEN, ...)
??? Don't get it ... |
Yes
Yes But I don't think that the reason is in a watchdog. The real reason is (from the healthy node):
Nope. I didn't stop corosync in this time. I started a normal cluster with two healthy nodes and hardly power off one of them from VirtualBox manager. The other one was rebooted with the last message in /var/log/messages as typed above.
Why not? Btw, I am not sure that the cores is virtual, because max number on cores (12) in the interface is equal to the number of hardware cores of CPU. With 2 cores for each two nodes the result is same. I powered off hardly the one healthy node and the other was rebooted: |
Just wanted to explain a way so that you don't get a node immediately rebooting after the timeout-issue is detected (meaning on the node where you stopped corosync). If you add a 3rd node to your cluster corosync-observation should work fine and the reboot of the node observed (or hard-shutdown) shouldn't tear down any peer-nodes. |
Nope, after I stopped corosync, the right node is rebooted. There is all fine.
Not "But", but "And". :) And we need that trick, because if the sbd is documented as a watchdog daemon, it must works as a watchdog daemon correctly. And all worked with option May be the problem, as I see in last log is: May be the problem is when corosync is in the state "ELECTION" it stops answering for a time.
Nope.
This is not my case. I don't have a 2-node-cluster, I have a 2-node-cluster-with-quorum-device and a 'real quorum' with 3 votes. |
|
To make this patch working with default timeouts, the default timeouts must be changed. Don't think that will be good to change timeouts of the corosync. IMHO will be better change timeouts in the sbd-inquisitor.c:574: The real timeout is the sum of the timeout_io (changed by Default timeout_loop is defined in sbd-common.c:44 And I think this will be better not touch. Default timeout_io is in sbd-common.c:45: So only need to change to Also need to change the man page. This is a full patch: |
|
I don't think it is a good idea that we try to cover up the symptoms instead of trying to find the reason. |
Okey, I like your desire to fix the reason and I'll help as I can.
Absolutely not. There is not any rt-trottling after I set 2 core per node. But problem still exists.
Nope. To achieve such result there must be 4s lag. But the lag is even more. Let me show you. I slightly change code of sbd-cluster.c in such way: And so: There is 27 seconds lag in the cpg_local_get(). Here is the full log.
This is very interesting. I don't know your setup. May be the matter in the new version of the corosync, CentOS 7 and RedHat 7 has very old soft. I'll try to compile a new version. PS I can compile corosync 3.0.1 and 2.99.5 on centos7, but I can't install. :) Well, no matter. As I said I like your patch as it is. |
Well I don't like it that much if it behaves like that on older corosync-versions ;-) ps: are you on clusterlabs-irc? just handier for chat ... pps: it seems that the reason of the difference in behavior of our setups isn't the corosync-version but rather use of qdevice :-( |
Why not? This can be described in the dependences or manual to set with old version of corosync option -I 30. |
That reduces responsiveness of the cluster by a factor of something like 5 or so. |
|
Main problem is because (for whatever reason) qdevice didn't provided ACK/NACK till watchdog was triggered - following line in the log is important: @Splarv Do you have an idea why qdevice decision takes too long (it shouldn't)? Some kind of slow heuristics? Something obvious in qdevice log? |
Bottom-line seems to be that the patch in here seems to be the right thing to do. |
Looked like a light at the end of a dark tunnel.
Okey, about qdevice. It is in the third VM with CentOS 7. Was installed by On the working node was configured by config on Witness VM Qdevice logs (I added |
|
@Splarv Thank you for super detailed logs! I was able to reproduce the scenario and now I finally fully understand what is happening, but I'm not entirely decided if it is bug or what we can do with that other than changing timeouts/document recommended timeouts (ether sbd or qdevice/qnetd). So what is happening:
So ~20 second timeout for sbd should be enough when used with qdevice (and for this IMHO kind of artificial, problem). |
It's frozen for clients, yes
Depends on token timeout. Default is 1000 + (number_of_nodes - 2) * 650.
It's not only about quorum (so one field). It's about sending/receiving cpg, quorum, cmap, cfg, ... This is more like if your db client would loose connection to sql server and db connector would return "random" (or unknown) values instead of waiting for TCP timeout - possible but not super useful. + It would break EVS what is just no no no (it's like if your psql would break ACID). Also there si a way how to check livenes even of corosync being in sync phase (calling votequorum) , but it actually doesn't help too much and only makes problem to appear later.
Would you sacrifice ACID? |
Nice enough. I saw only 1-2 second lag between Before cpg_local_get() and After true cpg_local_get(). I tried with the default timeout_io in sbd (3 seconds), the node with stopped corosync was rebooted in 3 seconds, while the healthy node raised masters and continue working without visible problems.
So for two nodes it will be... just 1 second. So remote nodes get a prove of life in 1 second, but local node got own the nodeid from the own corosync for 30 seconds (in worst case). Yep, as variant may be to change a way, how the local node gets a prove of life from the own corosync (do the same way, as do remote node). But with new timeouts for qnetd and qdevice all looking fine with default timeout_io of sbd. But what will be maximum theoretical lag for cpg_local_get()? If it will be 3 seconds (thus equal timeout_io of sbd) it is still dangerous becase it with small probability still will be able to suddenly reboot the last healthy node. I saw 3 seconds. May be need 4 to watchdog the last healthy node. It's unsafe, one 1 seconds lag and.... Is it depend on sync_timeout: of qdevice? I changed it to 2000. According to the manual:
May be even 1000 will be better here? |
As already stated above a delay of the time needed to get the quorum-state updated in here is exactly what we need. If we can't assume that quorum-state-updates come within the timeout we can't assume that a separated node is gonna suicide within this timeout. Consequence would be pacemaker introducing a split-brain. Atm I see issues with setting the qdevice-defaults that low as pcs doesn't support setting them up for one of the probably most used scenarios for qdevice (2 sites running services and a 3rd site - kind of poorly connected - as arbiter). |
As I can understand from the Also, timeouts 10s and 30s looked like qnetd was developed in the time of the phone modem internet. It will be not bad, if we move timeouts to the modern reality. |
|
And one more cent that the prove of life of the corosync must be slightly different. When I type Cib is not updated, the pacemaker show that the all fine, the PostgreSQL keep works in master mode (on both nodes: on the faulted old and is rised on healthy). In such situation the fault node must be watchdoged, as it will happens with working fencing. Afaik that is an issue with corosync 2.x and should be fixed in 3.x. |
|
@Splarv Thank you for the confirmation of working reduced timeouts. 10-30sec is not that much for "phone modem internet" (even that doesn't have such lags :) ) but mostly for heuristics which must also fit into this limit - and heuristics can be almost everything - including access to half-dead NFS server, ... Also we are talking about EVS so time itself (in theory) doesn't play any role (in practice it is quite opposite). Anyway, 10-30 sec is probably too much and when I was thinking about the problem little deeper I think it should be (by-default, it should stay fully configurable to whatever value one need) based on the token-timeout rather than some fixed value (like sync_timeout = 3*token_timeout) @wenningerk Could you please elaborate how you find out that "If we can't assume that quorum-state-updates come within the timeout we can't assume that a separated node is gonna suicide within this timeout."? Quorum-state-update will come within timeout (30 second is worst case with default config) and keep in mind all other nodes are blocked so there will be no split-brain - all nodes will wait for end of sync phase. "... verification that quorum-updates are coming in quick enough" - same question, why you need them in some time, when everything is synchronous (all nodes will block)? So why not to call votequorum (which has exception and calls proceed even in sync phase)? @Splarv |
I agree that the average roundtrip is never that much but WAN connections have hickups of a couple of seconds allowed in their contracts afaik.
Making it a multiple of token_timeout is a good idea I guess. There is anyway no sense in having a low token_timeout if quorum-updates are possibly delayed by way longer timeouts.
Wanted to state that it doesn't make sense here to require the corosync-test to return earlier than the maximum delay we have for quorum-state-updates.
Well it burns down to basic dealing with watchdogs. Have a small, easy to verify loop that triggers a hardware watchdog while it does heartbeating to other instances. Other instances have to adopt that principle in a way that either the mechanism used as heartbeating is gonna fail or they are operational with a very high probability - potentially breaking their own functionality again up according to a similar principle. |
Yep, I believe we are totally on same page in this respect. If somebody has a super slow wan then probably want a larger token timeout. If only qnetd is on super slow wan then they can set timeouts to whatever fits, but I believe 99.99% of use cases will work better with smaller timeouts. I will prepare some patch (hopefully soon). Said that it may make sense for sbd to also use such timeout (maybe 2 * sync_timeout?) so users become free of unexpected "surprise".
This is really pacemaker question, because when corosync unblocks (and all nodes unblocks in virtually same moment) it will inform both pacemaker and sbd and at least this message should proceed without any blocking.
Honestly I can imagine corosync to feed "watchdog" daemon (call it sbd/systemd-watchdogd/...) and such solution may be more reliable because we would remove "sampling" problem. But this still doesn't solve main problem. At least in corosync node itself never marks itself as dead. That is why I believe other nodes has to.
I know it is trade-of. But were you thinking about some kind of network poison pill? To really prevent situations when corosync becomes non-cooperative? |
There is some kind of chicken-and-egg issue here due to startup and stuff.
That takes us to a lack of a watchdog-model being usable by multiple instances.
Didn't think of much but network poison pill might be something to think of (depending on what one might see behind it ;-) ). Just wanted to deprive you from the idea that everything would be safe and sane in the poison-pill (on disk) case ;-) |
@jfriesse, can you explain "why?" such formulae. Or point me where I can read about it.
May be solution is to call votequorum, but check not that a quorum is quorable, but only that the return is without errors?
As I said there is situation when cluster is on different datacentres. And we try to solve the problems are concerned when one of DC is isolated or blackouted.
@jfriesse Thank you, you saved my time. Now I know that this is fixed in at least v2.99.3 (real commit is 96b4bd1).
Why not? Why sbd can not do what fencing do? |
I don't fully understand your question. If question is why it is larger than token timeout then idea is simple. Corosync changes membership because of something happened (node dead/switch dead/...) and it should give a qdevice/qnetd slightly more time to get all the heuristics done, possible slower connection to qnetd/... If it is other why then please be more specific what 'why' you mean.
Sure, this is why I've suggested
2.99.3 is pre-release of corosync which you probably do not want to use. Use 3.0.1, 2.4.4 or if you want not yet released version use camelback or needle branch.
I was talking about sbd without shared disk. That relies on Corosync/pacemaker being cooperative, what may not be always true (for various reasons, bugs included). As I've told X times during the thread, corosync never marks itself as dead. And also as Klaus described, everything what doesn't ack 'I've killed node-x successfully' is sort of sacrifice. |
Yep, why is bigger and why is bigger in 3 times. :) But not two, for instance.
Thanks for explanation. I am not so good in english and understanding of internal processes of pacemaker/corosync/etc to always understand what are you talking about. %(
I thought about v2.99.5, because v3.0.1. need even more dependences, for instance libknet, which is absent in CentOS 7. While 2.4.4 don't have this commit. Okey, I'll keep in mind to use 3.0.1 instead of 2.99.5.
As I understand you correctly, you talk about acknowledge 'I've killed node-x successfully'. In case of isolated sbd and watchdog this is not necessary, we just trust a watchdog. And this is not necessary because after successful watchdog, nobody will be to ask such acknowledge. :) Eh, may be you incorrectly understood me.
I mean only test for 'proof of life'. In case of fencing the remote node somehow get 'proof of live' from the other node and get decision to fence in case of the corosync or the pacemaker (don't know exactly who) of some node stop respond. Why sbd can't get the 'proof of life' in the exactly the same way? |
Can be. I mean, setting default timeouts is not exactly exact science. It's more like "I feel this may work". But does it mean it will fit everybody ? I would say "no way", but we must try to find something what is good for most of the people. And of course everything should be configurable so (this bug is nice example) you don't need to wait till I make a patch for qnetd/corosync/qdevice but you can just set whatever you need.
Yep, no worries.
Few comments:
Other nodes will ask.. Edit: Ok, They will not, but it's just because of
I'm not entirely sure if you don't see it other way around. Also I'm not entirely sure how exactly sbd works (edit: studied it little more so now I can understand how sbd works), but with standard power fencing it works following way:
SBD has no way for last step. There is nobody who can say that node was really turned off. edit: This is why sbd without disk needs @wenningerk Correct me if I'm totally wrong. |
|
@jfriesse Okey, my question was not philosophical. Let's return to your timeouts. You suggest token_timeout=1s, sync_timeout=3s. I see drawback of sync_timeout=3s, because it is equal to the default timeout_io in sbd and can occasionally fire a reboot by the inquisitor checking of servants. Can you point me is there will be obvious drawback of using token_timeout=1s, sync_timeout=2s (I didn't see) or token_timeout=1s, sync_timeout=1s? And with your timeouts the cluster reacts in ten times faster in the poweroff node case and I like this in any way.
I look forward to it.
I see your point: this do corosync, but we may not rely on local corosync to test local corosync. My question was how corosync tests other corosync? Does it send a special kind of package? Request? But I think now, that this check may be too complex to realise it outside corosync to check local corosync.
Yes. SBD use a watchdog/softdog and a default watchdog action, which is a hardware reset, usually. Or (or may be not 'or', but 'and') it use SysRq sequences, as I understand. Yep, relying on a watchdog is a risk, a watchdog can fail. But this is acceptable risk, a watchdog usually reliable enough and this is common to trust to it. So |
I've tried to explain it in comment #76 (comment) . Also I'm kind of expecting sbd will also adjust default timeout based on token timeout.
Very high level, corosync uses token ring to detect failures and order messages. For more info, you can check https://github.com/corosync/corosync/wiki/Developers#reference-documentation
|
Heh, I see slightly more time. :) Well, I tried investigate by myself. As I can understand there is a Well, you said that the sync_timeout must be slightly more time, because sync phase is a critical situation, when pacemaker and corosync do something, may be some heuristics (which is off by default) and a server has increased load. But modern servers are quick enough for pacemaker loads and so, as I see, there may be does not need additional time in sync phase. 1s may be sufficient. I checked, it works for me even on my 'slow' VMs.
Thanks for point me to the right direction. The documentation was useless, I looked into the source. I am interesting in |
|
@Splarv Honestly it looks like you are grasping the problem only from your perspective of two node cluster (probably not super overloaded). I'm grasping the problem from wider perspective, where 2 node cluster is super popular, but some customers demands 32 nodes (and timeout there is really not 1sec). Perspective when sometimes machines are so overloaded that corosync is not scheduled for 3+ seconds. So I think 3 * token_timeout is going to be good default value. If you think that 1 s will fit your deployment better you have possibility to change timeouts in config. |
|
I guess the main risk in using a watchdog isn't the reliability of the watchdog itself. That should be sufficiently high. |
|
@wenningerk I believe you are still considering setting default timeout based on token timeout, are you? Because otherwise no matter what I will set as a default we get into problems with higher number of nodes (14 nodes = default timeout 10.1s). Also it's not 3s. Keep in mind that timeout starts on both other nodes and qnetd side, so real timeout is actually sync_timeout - token_timeout = 2s (for two node cluster). |
|
@jfriesse: atm approach is that pacemaker checks if stonith-watchdog-timeout is compatible with (bigger) than watchdog-timeout configured with sbd. On a node where this isn't fulfilled pacemaker isn't gonna come up and start resources and thus there isn't gonna be any problem with that node not rebooting quickly enough. |
|
@wenningerk Ok, I believe we are on same page. "largest sbd-timeout in the cluster" seems (IMHO) as a best and most reliable way. I'm not entirely sure if corosync and sbd should be coming up simultaneously, but if it should then what about having some "default" timeout which will be used until better (defined by corosync/shared-disk/....) is available? Anyway. I was thinking about all consequences of timeout running on both qnetd side and other nodes side and I believe sbd will work even without changing its defaults for large clusters, because when other nodes finally find out that victim node is dead, qnetd will already know it. So reply will be instant (with exception of slow heuristics). |
Well, I think that the default configuration must be for most popular cases and unexperienced users, but not for very rare cases with very professional admins. :) But no matter, I see that @wenningerk exactly said my point about 3s.
And you already discussed about this somehow. Now I can confirm that I was understood. :) |
|
Okey, comrades. I need your advise and may be help. I tried, but I don't see easy way to get totem_stats_t.hdr.last_updated in the sbd. But looked like I can use for a heartbeat the orf_token_rx from the cmap. It constantly increased in the working node, but on an node isolated by The first question: "Will this be correct?" And say the truth it will be not easy for me. As I see there is already exist some logic for cmap, for instance to connect to the cmap service, but only in the conditional compilation directives CHECK_TWO_NODE. And so it is hard for me to change the code with keeping in mind all caveats of conditional compilation, which I don't know. |
|
I don't think so, because cmap call is blocked during sync phase same way as cpg is (for same good reason) - or maybe I don't understand the idea (I have exactly zero experience with sbd code). |
Yep, it's not bad, may be, with right timeouts. The reason of my idea is to make not a trivial check but check the health of corosync. It may help in cases when corosync looked like working, but don't work indeed. For instance, in the case of |
|
Perhaps after PR#83 is accepted, this PR may be closed. |
I've opened https://bugzilla.redhat.com/show_bug.cgi?id=1703128 to not loose track of the more sophisticated checking. |
4 participants
No description provided.