Skip to content

Handle unknown runtime exception while reading entries#2993

Merged
rdhabalia merged 2 commits into
apache:masterfrom
rdhabalia:read_catch
Nov 16, 2018
Merged

Handle unknown runtime exception while reading entries#2993
rdhabalia merged 2 commits into
apache:masterfrom
rdhabalia:read_catch

Conversation

@rdhabalia

@rdhabalia rdhabalia commented Nov 15, 2018

Copy link
Copy Markdown
Contributor

Motivation

Recently we have seen below runtime-exception while reading entries at broker and broker is not handling it correctly and not completing read callback so, dispatch get stuck for the subscription.

22:16:58.424 [bookkeeper-ml-workers-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught 
java.lang.NullPointerException: null
        at org.apache.bookkeeper.mledger.impl.EntryImpl.create(EntryImpl.java:89) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.EntryCacheImpl.asyncReadEntry(EntryCacheImpl.java:225) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.internalReadFromLedger(ManagedLedgerImpl.java:1509) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncReadEntries(ManagedLedgerImpl.java:1359) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.notifyEntriesAvailable(ManagedCursorImpl.java:2221) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.lambda$notifyCursors$27(ManagedLedgerImpl.java:1580) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [bookkeeper-common-4.7.2.jar:4.7.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.22.Final.jar:4.1.22.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

3:22:26.425 [bookkeeper-ml-workers-OrderedExecutor-10-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught 
java.lang.NullPointerException: null
        at org.apache.bookkeeper.mledger.impl.EntryImpl.create(EntryImpl.java:89) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.EntryCacheImpl.asyncReadEntry(EntryCacheImpl.java:225) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.internalReadFromLedger(ManagedLedgerImpl.java:1509) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncReadEntries(ManagedLedgerImpl.java:1359) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.OpReadEntry.lambda$checkReadCompletion$76(OpReadEntry.java:136) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]


        06:58:57.334 [bookkeeper-ml-workers-OrderedExecutor-19-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught 
java.lang.NullPointerException: null
        at org.apache.bookkeeper.mledger.impl.EntryImpl.create(EntryImpl.java:89) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.EntryCacheImpl.asyncReadEntry(EntryCacheImpl.java:225) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.internalReadFromLedger(ManagedLedgerImpl.java:1509) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncReadEntries(ManagedLedgerImpl.java:1359) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.notifyEntriesAvailable(ManagedCursorImpl.java:2221) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.lambda$notifyCursors$27(ManagedLedgerImpl.java:1580) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [bookkeeper-common-4.7.2.jar:4.7.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [pulsar-functions-metrics-2.2.2-yahoo.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

Modifications

Handle NPE and complete the callback with failure.

Result

Dispatch will not be blocked if there is runtime exception while reading entries.

@rdhabalia rdhabalia added this to the 2.2.1 milestone Nov 15, 2018
@rdhabalia rdhabalia self-assigned this Nov 15, 2018
@rdhabalia
rdhabalia requested a review from merlimat November 15, 2018 21:47
merlimat
merlimat reviewed Nov 15, 2018
View reviewed changes
Comment thread managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/EntryCacheImpl.java Outdated
@merlimat merlimat added the type/bug The PR fixed a bug or issue reported a bug label Nov 15, 2018
@merlimat

merlimat commented Nov 15, 2018

Copy link
Copy Markdown
Contributor

The suspicion is that this might be due to the scenario described in netty/netty#8563

merlimat
merlimat approved these changes Nov 15, 2018
View reviewed changes

@merlimat merlimat left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@merlimat merlimat mentioned this pull request Nov 16, 2018
Merged
@rdhabalia
rdhabalia merged commit 2c8c288 into apache:master Nov 16, 2018
merlimat pushed a commit that referenced this pull request Dec 13, 2018
@rdhabalia @merlimat
Handle unknown runtime exception while reading entries (#2993)
61ce27d 
* Handle unknown runtime exception while reading entries

* make asyncReadEntry0 private
@lhotari lhotari mentioned this pull request May 3, 2021
Closed
@mattisonchao mattisonchao mentioned this pull request Nov 1, 2022
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@merlimat merlimat merlimat approved these changes
@massakam massakam massakam approved these changes

Assignees

@rdhabalia rdhabalia

Labels

area/broker type/bug The PR fixed a bug or issue reported a bug

Projects

None yet

Milestone

2.2.1

Development

Successfully merging this pull request may close these issues.

3 participants

@rdhabalia @merlimat @massakam