Add more config for auto-topic-creation by fxbing · Pull Request #4963 · apache/pulsar

fxbing · 2019-08-15T15:00:50Z

Master Issue: #4926

Motivation

Curently the partitioned-topic and non-partitioned topic is a little confuse for users. in PR #3450 we add config for auto-topic-creation.
We could leverage this config to provide some more config for auto-topic-creation.

Modifications

Add allowAutoTopicCreationType and allowAutoTopicCreationNumPartitions to configuration.
Users can use both configurations when they decide to create a topic automatically.
Add test.
Update doc.

sijie

@fxbing it is a nice motivation. but I am feeling that you didn't change the code in the right place. the change in fetchPartitionedTopicMetadataAsync will introduce unnecessary creation of partitioned topic. The auto-creation happens when a producer or a consumer connects to broker. so I think the change should be done in BrokerService. You can check the methods getOrCreateTopic.

Also change "partition" and "non-partition" to "partitioned" and "non-partitioned". It might be also worth considering adding an enum like TopicType?

fxbing · 2019-08-16T04:11:25Z

@fxbing it is a nice motivation. but I am feeling that you didn't change the code in the right place. the change in fetchPartitionedTopicMetadataAsync will introduce unnecessary creation of partitioned topic. The auto-creation happens when a producer or a consumer connects to broker. so I think the change should be done in BrokerService. You can check the methods getOrCreateTopic.

I first thought about making changes in the BrokerService, but when the client requests to create a Producer or Consumer, it will first get partitions info from zk, decide whether to create partitioned or non-partitioned. When the request arrives at the BrokerService, the client has decided to create a nonpartitioned Producer or Consumer. If the Broker's configuration requires automatic creation of a partitioned topic, Broker needs to send the configuration information to client, client re-establishes the partitioned Producer or Consumer, and the BrokerService updates the partitions in the zk.

I think it's easier to make changes when the client creates the type that determines the creation, and need fewer communications.

Do I need to modify it as I said above?

fxbing · 2019-08-16T04:13:18Z

Also change "partition" and "non-partition" to "partitioned" and "non-partitioned". It might be also worth considering adding an enum like TopicType?

OK， I will make a change.

fxbing · 2019-08-17T12:16:36Z

run java8 tests

fxbing · 2019-08-18T01:22:52Z

run Integration Tests

sijie

@fxbing well done. I have two questions to clarify with you and @jiazhai

the first one is about the setting name about the number of partitions.
the second one is whether we need to check if a non-partitioned topic exists or not when the partitioned-topic doesn't exist.

PTAL

fxbing · 2019-08-20T09:58:03Z

run C++ / Python Tests
run Integration Tests

fxbing · 2019-08-20T09:59:34Z

run cpp tests

fxbing · 2019-08-22T06:00:46Z

run cpp tests

fxbing · 2019-08-22T06:51:33Z

run cpp tests
run integration tests
run java8 tests

fxbing · 2019-08-22T09:47:44Z

run cpp tests
run integration tests
run java8 tests

fxbing · 2019-08-25T04:15:11Z

run cpp tests
run integration tests
run java8 tests

fxbing · 2019-09-11T14:49:17Z

run integration tests

fxbing · 2019-09-12T00:30:15Z

run java8 tests

fxbing · 2019-09-12T03:01:29Z

PTAL @sijie @jiazhai @codelipenghui

codelipenghui

@fxbing Nice work!
Overall looks good to me, just left a suggestion

codelipenghui · 2019-09-17T06:40:09Z

+        return metadataFuture;
+    }
+
+    protected static CompletableFuture<PartitionedTopicMetadata> createDefaultPartitionedTopicAsync(


Since this method is no longer an asynchronous method, it's better to rename it to createDefaultPartitionedTopic()

sijie · 2019-09-17T14:01:31Z

@jiazhai @jennifer88huang can you please make sure this feature is well documented?

Jennifer88huang-zz · 2019-09-17T14:27:38Z

Got it.

Master Issue: #4926 Curently the partitioned-topic and non-partitioned topic is a little confuse for users. in PR #3450 we add config for auto-topic-creation. We could leverage this config to provide some more config for auto-topic-creation. - Add `allowAutoTopicCreationType` and `allowAutoTopicCreationNumPartitions` to configuration. - Users can use both configurations when they decide to create a topic automatically. - Add test. - Update doc. (cherry picked from commit 547c421)

sijie requested changes Aug 16, 2019

View reviewed changes

fxbing mentioned this pull request Aug 18, 2019

org.apache.pulsar.tests.integration.functions.PulsarFunctionsThreadTest failure #4973

Closed

sijie reviewed Aug 18, 2019

View reviewed changes

Comment thread conf/broker.conf Outdated

Comment thread pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java Outdated

sijie assigned fxbing Aug 18, 2019

sijie added area/admin area/broker type/feature The PR added a new feature or issue requested a new feature labels Aug 18, 2019

sijie added this to the 2.5.0 milestone Aug 18, 2019

sijie mentioned this pull request Aug 18, 2019

Add more config for auto-topic-creation #4926

Closed

jiazhai requested changes Aug 20, 2019

View reviewed changes

Comment thread pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java Outdated

Comment thread pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java Outdated

fxbing force-pushed the add-config-for-auto-topic-creation branch from 5e7a633 to 0877f77 Compare August 20, 2019 07:32

This was referenced Aug 20, 2019

BinaryLookupServiceTest.basicLookup failed #4985

Closed

AdminMultiHostTest, CLITest, HealthcheckTest and PulsarFunctionsProcessTest failed #4984

Closed

fxbing force-pushed the add-config-for-auto-topic-creation branch 2 times, most recently from c4382da to de35875 Compare August 22, 2019 03:32

fxbing force-pushed the add-config-for-auto-topic-creation branch 5 times, most recently from 580d8a7 to a068ba3 Compare August 28, 2019 06:45

fxbing added 10 commits September 11, 2019 20:24

fix test

41fa7ca

create async method for default partitioned topic creation

87dd938

fix comment

a38efb3

fix comment

30913de

fix comment

62f2313

fix comment

65978c6

add test

84bd59b

fix test

ee69a44

fix test

0408e34

fix test

fd5da63

fxbing force-pushed the add-config-for-auto-topic-creation branch from c2e0629 to fd5da63 Compare September 11, 2019 12:24

fxbing requested review from codelipenghui, jiazhai and sijie September 12, 2019 03:00

jiazhai approved these changes Sep 12, 2019

View reviewed changes

codelipenghui approved these changes Sep 17, 2019

View reviewed changes

sijie approved these changes Sep 17, 2019

View reviewed changes

sijie merged commit 547c421 into apache:master Sep 17, 2019

Anonymitaet mentioned this pull request Sep 19, 2019

[Doc] Add configs for broker #5223

Closed

This was referenced Oct 3, 2019

Ensure the handling of PartitionMetadataRequest is async end-to-end #5307

Merged

Revert default topic type to non-partitioned #5308

Merged

wolfstudy modified the milestones: 2.5.0, 2.4.2 Nov 22, 2019

BewareMyPower mentioned this pull request Nov 18, 2022

Fix getting partition metadata of a nonexistent topic returns 0 #10601

Merged

1 task

Uh oh!

Conversation

fxbing commented Aug 15, 2019

Motivation

Modifications

Uh oh!

sijie left a comment

Choose a reason for hiding this comment

Uh oh!

fxbing commented Aug 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fxbing commented Aug 16, 2019

Uh oh!

fxbing commented Aug 17, 2019

Uh oh!

fxbing commented Aug 18, 2019

Uh oh!

sijie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fxbing commented Aug 20, 2019

Uh oh!

fxbing commented Aug 20, 2019

Uh oh!

fxbing commented Aug 22, 2019

Uh oh!

fxbing commented Aug 22, 2019

Uh oh!

fxbing commented Aug 22, 2019

Uh oh!

fxbing commented Aug 25, 2019

Uh oh!

fxbing commented Sep 11, 2019

Uh oh!

fxbing commented Sep 12, 2019

Uh oh!

fxbing commented Sep 12, 2019

Uh oh!

codelipenghui left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codelipenghui Sep 17, 2019

Choose a reason for hiding this comment

Uh oh!

sijie commented Sep 17, 2019

Uh oh!

Jennifer88huang-zz commented Sep 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

fxbing commented Aug 16, 2019 •

edited

Loading

codelipenghui left a comment •

edited

Loading