Search before asking
Read release policy
Version
pulsar server: docker image apachepulsar/pulsar:3.0.4 + helm chart pulsar-helm-chart
pulsar client: java client org.apache.pulsar:pulsar-client:3.0.4
Minimal reproduce step
After updating Apache Pulsar, we noticed that one of the consumers sometimes stops receiving new messages for some topics.
The last fully working version for us is 3.0.1. I have tested all later versions released so far and also built a branch-3.0.
I looked through the commits and determined when our service stops working:
- Last commit where our service works properly: 80a8f8d
- Commit which breaks, our service no longer works properly: 6e59208
I performed a test using the last commit from branch 3.0 (fd823f6) and reverting the individualAckNormal method to the last version before the "commit which breaks." The change looks as follows: 180254@6dac4bf. I have no problem with the modified code.
I found nothing in the logs that would inform me about the consumer suspension, etc. There are no unusual logs at all. Restarting the Kubernetes pod with consumers has helped for some time.
What did you expect to see?
consumer retrieves all messages
What did you see instead?
consumers stops receiving new messages for some topics
Anything else?
The configuration we use:
broker:
managedLedgerDefaultEnsembleSize: "3"
managedLedgerDefaultWriteQuorum: "3"
managedLedgerDefaultAckQuorum: "2"
brokerDeduplicationEnabled: "true"
bookkeeperClientTimeoutInSeconds: "5"
bookkeeperClientHealthCheckErrorThresholdPerInterval: "3"
bookkeeperClientHealthCheckQuarantineTimeInSeconds: "600"
bin/pulsar-admin --admin-url "${ADMIN_URL}" namespaces create "${TENANT}/service"
bin/pulsar-admin --admin-url "${ADMIN_URL}" namespaces set-max-unacked-messages-per-consumer -c 10 "${TENANT}/service"
bin/pulsar-admin --admin-url "${ADMIN_URL}" namespaces set-max-unacked-messages-per-subscription -c 20 "${TENANT}/service"
persistent://public/service/service12_some_topic_someotherpart
- we use PatternMultiTopicsConsumerImpl
return pulsarClient
.newConsumer(Schema.STRING)
.subscriptionName(pulsarServerBasename)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.subscriptionType(SubscriptionType.Shared)
.topicsPattern(Pattern.compile("persistent://public/service/service12_.+"))
.negativeAckRedeliveryDelay(1000, TimeUnit.MILLISECONDS)
.patternAutoDiscoveryPeriod(60, TimeUnit.SECONDS)
.receiverQueueSize(1)
We can reproduce it on our service. Test scenario: serviced approximately 20 customers (== 20 topics), each with about 20 messages per second. 1 message is processed in approximately 200ms. The problem occurs for a certain number of topics in the test, not for all
When a problem occurs:
- pulsar_subscription_back_log metric shows that the backlog is growing
- pulsar_subscription_unacked_messages metric shows 0
- pulsar_subscription_blocked_on_unacked_messages metric shows 0
- in the service metrics (consume & process pod), I do not see that any messages for the broken topic are being processed
Are you willing to submit a PR?
Search before asking
Read release policy
Version
pulsar server: docker image apachepulsar/pulsar:3.0.4 + helm chart pulsar-helm-chart
pulsar client: java client org.apache.pulsar:pulsar-client:3.0.4
Minimal reproduce step
After updating Apache Pulsar, we noticed that one of the consumers sometimes stops receiving new messages for some topics.
The last fully working version for us is 3.0.1. I have tested all later versions released so far and also built a branch-3.0.
I looked through the commits and determined when our service stops working:
I performed a test using the last commit from branch 3.0 (fd823f6) and reverting the individualAckNormal method to the last version before the "commit which breaks." The change looks as follows: 180254@6dac4bf. I have no problem with the modified code.
I found nothing in the logs that would inform me about the consumer suspension, etc. There are no unusual logs at all. Restarting the Kubernetes pod with consumers has helped for some time.
What did you expect to see?
consumer retrieves all messages
What did you see instead?
consumers stops receiving new messages for some topics
Anything else?
The configuration we use:
We can reproduce it on our service. Test scenario: serviced approximately 20 customers (== 20 topics), each with about 20 messages per second. 1 message is processed in approximately 200ms. The problem occurs for a certain number of topics in the test, not for all
When a problem occurs:
Are you willing to submit a PR?