We have a very strange issue with partitioned topics and shared_key subscriptions, every few days no messages are being consumed from some of the partitions and the backlog keeps growing, the issue persists until we resubscribe or unloading the topic.
pulsar version is 2.6.0
you can see the issue in this screenshot partition 0 and 2 (most left and most right) are stuck, while partition 1 is consumed without any issues.
I saw the following Exception in the brokers around the last consumed/acked time (edited)
not sure if it's related, couldn't see any errors on the clients.
2020-08-07T04:37:03.485 | java.lang.ArrayIndexOutOfBoundsException: null
04:37:03.484 [bookkeeper-ml-workers-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught
04:37:03.478 [pulsar-io-23-3] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.130.2.20:50286] Created subscription on topic persistent://internal/emsaas/enriched-event-unified-partition-0 / property-worker-cloud
We have a very strange issue with partitioned topics and shared_key subscriptions, every few days no messages are being consumed from some of the partitions and the backlog keeps growing, the issue persists until we resubscribe or unloading the topic.
pulsar version is 2.6.0
you can see the issue in this screenshot partition 0 and 2 (most left and most right) are stuck, while partition 1 is consumed without any issues.
I saw the following Exception in the brokers around the last consumed/acked time (edited)
not sure if it's related, couldn't see any errors on the clients.
2020-08-07T04:37:03.485 | java.lang.ArrayIndexOutOfBoundsException: null
04:37:03.484 [bookkeeper-ml-workers-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught
04:37:03.478 [pulsar-io-23-3] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.130.2.20:50286] Created subscription on topic persistent://internal/emsaas/enriched-event-unified-partition-0 / property-worker-cloud