[fix] [tx] [branch-2.9] Transaction buffer recover blocked by readNext

[poorbarcode]

Motivation

Since PR #18833 can not cherry-pick to branch-2.9, create a separate PR.

Context for Transaction Buffer

• If turn on transactionCoordinatorEnabled, then TransactionBuffer will be initialized when a user topic create.
• The TransactionBuffer reads the aborted logs of transactions from topic __transaction_buffer_snapshot -- this process is called recovery.
• During recovery, the reading from that snapshot ledger is done via a Reader; the reader works like this:

while (reader.hasMessageAvailable()){
    reader.readNext();
}

Context for Compaction

• After pip-14, the consumer that enabled feature read-compacted will read messages from the compacted topic instead of the original topic if the task-compaction is done, and read messages from the original topic if task-compaction is not done.
• If the data of the last message with key k sent to a topic is null, the compactor will mark all messages for that key as deleted.

Issue

There is a race condition: after executing reader.hasMessageAvailable, the following messages have been deleted by compaction-task, so read next will be blocked because there have no messages to read.

---

Modifications

• If hits this issue, do recover again.

---

Why not just let the client try to load the topic again to retry the recover?

If the topic load is failed, the client will receive an error response. This is a behavior that we can handle, so should not be perceived by the users.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

• [ci] [fix] [tx] Transaction buffer recover can not completed poorbarcode/pulsar#48

[congbobo184]

should close the reader right?

[poorbarcode]

Yes, Already fixed.