Entry duplicated in pulsar when bookie failed and zookeeper disconnected

[kezhuw]

Describe the bug
When adding new entry, bookie failure and zookeeper disconnection may cause inconsistency between LedgerInfo.getEntries and LedgerMetadata.getLastEntryId. This will introduce duplicated entry.

To Reproduce

1. Adding entry-a to ledger1.
2. Succeed to persist on disk, but failed to response due to, say, machine crash.
3. Zookeeper disconnected, thus fail to write metadata and closing ledger.
4. Report ledgerClosed to ManagedLedger.
5. Zookeeper reconnected.
6. Roll to ledger2, entry-a added success.
7. ManagedLedger does not count entry-a in LedgerInfo.getEntries, but entry-a does count as LedgerMetadata.getLastEntryId and LedgerHandle.getLastAddConfirmed after recovery.
8. ManagedCursor does not use LedgerInfo.getEntries to restrict its reading.
9. LedgerOffloader does not use LedgerInfo.getEntries either.

I have add test case to reproduce this: kezhuw@bc0de5e.

Strictly speaking, I think OpAddEntry.handleAddTimeoutFailure has this issue too.

[kezhuw]

Forgot to attach test case assertion message for kezhuw@bc0de5e, here it is:

java.lang.AssertionError: The following asserts failed:
	unexpected last entry id expected [9] but found [10],
	unexpected last add confirmed expected [9] but found [10],
	unexpected sequence id expected [12] but found [11],
	unexpected message content expected [foo-12] but found [foo-11],
	unexpected sequence id expected [13] but found [12],
	unexpected message content expected [foo-13] but found [foo-12]
<Click to see difference>


	at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:43)
	at org.apache.pulsar.client.api.ClientDeduplicationFailureTest.testClientDeduplicationWithBkFailureAndZkDisconnected(ClientDeduplicationFailureTest.java:593)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
	at org.testng.TestRunner.privateRun(TestRunner.java:648)
	at org.testng.TestRunner.run(TestRunner.java:505)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
	at org.testng.SuiteRunner.run(SuiteRunner.java:364)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
	at org.testng.TestNG.runSuites(TestNG.java:1049)
	at org.testng.TestNG.run(TestNG.java:1017)
	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:110)

[tisonkun]

Closed as stale. Please create a new issue if it's still relevant to the maintained versions.