ZOOKEEPER-4785: Txn loss due to race condition in Learner.syncWithLeader() during DIFF sync

[li4wang]

Provides a fix for the txn loss issue caused by the race condition in DIFF sync.

Author: Li Wang liwang@apple.com

[eolivelli]

LGTM

nice catch !

[ChengYilong89]

hello, my friend. I met a issue in the production, the temporary node in the lead is deleted, however the temporary node in the other followers is not deleted. so we got data inconsistent between lead and follower. the version of ZK is 3.8.1.
Then I saw the ISSUE-4785, and my scenario is similar with the ISSUE-4785, the follower A sync with the leader, and old leader restart, and the follower A became a new leader.
Does ISSUE-4785 can caused the data inconsistent? In your description, what does "5. The follower was bounced before writing all the uncommitted txns to disk" means? @li4wang

[li4wang]

Does ISSUE-4785 can caused the data inconsistent?

Data inconsistence here means digest mismatch count is greater than 0 or you mean client see different data from leader and follower?

what does "5. The follower was bounced before writing all the uncommitted txns to disk"

This means that follower didn't persist the txns from diff sync with leader into disk before shutdown and the follower lost the txns.

[ChengYilong89]

Data inconsistence here means digest mismatch count is greater than 0 or you mean client see different data from leader and follower?
---- I mean client see different data from leader and follower. for example, leader contains data of A, B, C, but the follower can only see B, C. the A node is a temporary, and it should be deleted and sync to the follower. But in fact, it is not deleted in the follower.

This means that follower didn't persist the txns from diff sync with leader into disk before shutdown and the follower lost the txns.
----- can this result in data inconsistent between leader and follower?

@li4wang