From b01948d0ebd2bca563fb3b6a815f1ebb9640802d Mon Sep 17 00:00:00 2001
From: Nikolai Borisov <nikolai.borisov@onde.app>
Date: Thu, 23 Nov 2023 17:12:42 +0400
Subject: [PATCH] [improve][pip] PIP-319 Unblock stuck Key_Shared subscription
 after consumer reconnect

---
 pip/pip-319.md | 101 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 101 insertions(+)
 create mode 100644 pip/pip-319.md

diff --git a/pip/pip-319.md b/pip/pip-319.md
new file mode 100644
index 0000000000000..0965774fec781
--- /dev/null
+++ b/pip/pip-319.md
@@ -0,0 +1,101 @@
+# PIP-319: Unblock stuck Key_Shared subscription after consumer reconnect
+
+# Background knowledge
+
+Key_Shared subscriptions has guarantees that all the messages with the same key will be delivered in the order they were sent (https://pulsar.apache.org/docs/3.1.x/concepts-messaging/#key_shared)
+
+Current delivery mechanism blocks recently joined consumers from getting messages until all the previously delivered but not acknowledged messages are either acknowledged by the consumer which received the messages or the consumer disconnected
+
+This is implementation trade off which allows not to track all the sent messages. This approach allows to reduce memory consumption and increase performance
+
+The same time such implementation leads to a situation when a single failed message could lead to all the consumers blockage (described at https://github.com/apache/pulsar/issues/21199)
+
+There is a setting `allowOutOfOrderDelivery` which allows to mitigate the described issue but leads to ordering guarantees loss
+
+
+# Motivation
+
+For some scenarios both consumption stuck and out of order delivery is not an option. The same time it could be ok to have more memory consumption and some minor performance degradation
+
+# Goals
+
+The purpose of this PIP is to introduce a configuration option which allow to track all the messages which are not acknowledged
+
+This option will allow to unblock stuck Key_Shared subscriptions for recently joined consumers introducing memory and performance overhead 
+
+
+## In Scope
+
+- Add broker configuration to track not acknowledged messages
+- Implement tracking not acknowledged messages and avoiding recently joined consumers blockage for key_share subscriptions
+
+
+# High Level Design
+
+Add `shouldRememberUnackedMessageKey` and if enabled start tracking all the messages which were sent to consumer but were not acknowledged
+
+If `shouldRememberUnackedMessageKey` enabled send messages with keys which were not sent to existing consumers
+
+This will allow to recently joined consumers not to wait until all the previously sent messages are consumed
+
+# Detailed Design
+
+## Design & Implementation Details
+
+1. Add `shouldRememberUnackedMessageKey` setting to broker.conf
+2. Add `pendingMessages` map to track all the messages which were sent to a consumer but not acknowledged to `org.apache.pulsar.broker.service.persistent.PersistentSubscription`
+3. If the setting `shouldRememberUnackedMessageKey` enabled add messages to the field `pendingMessages` after a message sent
+4. If the setting `shouldRememberUnackedMessageKey` enabled remove messages from the field `pendingMessages` after a message acknowledged 
+5. If the setting `shouldRememberUnackedMessageKey` enabled clear the field `pendingMessages` when `org.apache.pulsar.broker.service.Consumer.clearUnAckedMsgs` method invoked
+6. If the setting `shouldRememberUnackedMessageKey` enabled remove deleted consumer messages from the field `pendingMessages` when `org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers.removeConsumer` method invoked
+7. If the setting `shouldRememberUnackedMessageKey` enabled allow sending messages which keys not present at `pendingMessages` field in `org.apache.pulsar.broker.service.persistent.PersistentStickyKeyDispatcherMultipleConsumers`
+
+
+## Public-facing Changes
+
+### Public API
+
+### Binary protocol
+
+### Configuration
+
+Broker will have new configuration option `shouldRememberUnackedMessageKey` in broker.conf:
+
+```
+# If enabled subscriptions stores keys of messages which allows consumer not
+# to stuck in case it goes to recently assigned. The setting allows to overcome
+# situation when new KeyShared consumers will not get any messages until a consumer
+# that did get messages disconnects or acks/nacks some messages
+# The trade of is the need to track all the not acked messages in subscription
+# which could potentially lead to performance degradation and higher memory consumption
+rememberNotAckedMessagesKey=false
+```
+
+### CLI
+
+### Metrics
+
+# Monitoring
+
+
+# Security Considerations
+
+# Backward & Forward Compatibility
+
+## Revert
+
+Reverting to a previous version will simply get rid of this config/limitation which is the previous behavior.
+
+## Upgrade
+
+# Alternatives
+
+# General Notes
+
+# Links
+
+<!--
+Updated afterwards
+-->
+* Mailing List discussion thread:
+* Mailing List voting thread: