Add topic operation commands by zymap · Pull Request #56 · streamnative/pulsarctl

zymap · 2019-09-13T16:11:09Z

compact

➜  pulsarctl git:(topic-operation) ./pulsarctl topic compact -h
USED FOR:
    This command is used for compacting a persistent topic.

REQUIRED PERMISSION:
    This command is requires tenant admin permissions.

EXAMPLES:
    #Compact a persistent topic <topic-name>
    pulsarctl topic compact <topic-name>

OUTPUT:
    #normal output
    Sending compact topic <topic-name> request successfully

    #the topic name is not specified
    [✖]  only one argument is allowed to be used as a name

    #the specified topic does not found
    [✖]  code: 404 reason: Topic not found

    #the topic name is not in the format of <tenant>/<namespace>/<topic> or <topic>
    [✖]  Invalid short topic name '<topic-name>', it should be in the format of <tenant>/<namespace>/<topic> or <topic>

    #the topic name is not in the format of <domain>://<tenant>/<namespace>/<topic>
    [✖]  Invalid complete topic name '<topic-name>', it should be in the format of <domain>://<tenant>/<namespace>/<topic>

    #the topic name is not in the format of <tenant>/<namespace>/<topic>
    [✖]  Invalid topic name '<topic-name>', it should be in the format of<tenant>/<namespace>/<topic>

    #the namespace name is not in the format of <tenant>/<namespace>
    [✖]  The complete name of namespace is invalid. complete name : <namespace-complete-name>

    #the tenant name and(or) namespace name is empty
    [✖]  Invalid tenant or namespace. [<tenant>/<namespace>]

    #the tenant name contains unsupported special chars. the alphanumeric (a-zA-Z0-9) and the special chars (-=:.%)  is allowed
    [✖]  Tenant name include unsupported special chars. tenant : [<namespace>]

    #the namespace name contains unsupported special chars. the  alphanumeric (a-zA-Z0-9) and the special chars (-=:.%) is allowed
    [✖]  Namespace name include unsupported special chars. namespace : [<namespace>]

Usage: pulsarctl topics compact [flags]

compact-status

➜  pulsarctl git:(topic-operation) ./pulsarctl topic compact-status -h
USED FOR:
    This command is used for getting status of compaction on a topic.

REQUIRED PERMISSION:
    This command requires tenant admin permissions.

EXAMPLES:
    #Get status of compaction of a persistent topic <topic-name>
    pulsarctl topic compact-status <topic-name>

OUTPUT:
    #normal output
    Compaction was a success

    #Compaction is not running
    Compaction has not been run for <topic-name> since broker startup

    #Compaction is running
    Compaction is currently running

    #Compaction is error
    Error in compaction

    #the topic name is not specified
    [✖]  only one argument is allowed to be used as a name

    #the specified topic does not found
    [✖]  code: 404 reason: Topic not found

    #the topic name is not in the format of <tenant>/<namespace>/<topic> or <topic>
    [✖]  Invalid short topic name '<topic-name>', it should be in the format of <tenant>/<namespace>/<topic> or <topic>

    #the topic name is not in the format of <domain>://<tenant>/<namespace>/<topic>
    [✖]  Invalid complete topic name '<topic-name>', it should be in the format of <domain>://<tenant>/<namespace>/<topic>

    #the topic name is not in the format of <tenant>/<namespace>/<topic>
    [✖]  Invalid topic name '<topic-name>', it should be in the format of<tenant>/<namespace>/<topic>

    #the namespace name is not in the format of <tenant>/<namespace>
    [✖]  The complete name of namespace is invalid. complete name : <namespace-complete-name>

    #the tenant name and(or) namespace name is empty
    [✖]  Invalid tenant or namespace. [<tenant>/<namespace>]

    #the tenant name contains unsupported special chars. the alphanumeric (a-zA-Z0-9) and the special chars (-=:.%)  is allowed
    [✖]  Tenant name include unsupported special chars. tenant : [<namespace>]

    #the namespace name contains unsupported special chars. the  alphanumeric (a-zA-Z0-9) and the special chars (-=:.%) is allowed
    [✖]  Namespace name include unsupported special chars. namespace : [<namespace>]

Usage: pulsarctl topics compact-status [flags]

offload

➜  pulsarctl git:(topic-operation) ./pulsarctl topic offload -h
USED FOR:
    This command is used for triggering offload the data from a topic to long-term storage (e.g. Amazon S3)

REQUIRED PERMISSION:
    This command requires tenant admin permissions.

EXAMPLES:
    #Trigger offload the data from a topic <topic-name> to long-term storage and keep <threshold> size data in BookKeeper (e.g. 10M, 5G, default is byte)
    pulsarctl topic offload <topic-name> <threshold>

OUTPUT:
    #normal output
    Offload trigger for <topic-name> for messages before <message-id>

    #noting to offload
    Nothing to offload

    #the topic name is not specified or the offload threshold is not specified
    [✖]  only two argument is allowed to be used as names

    #the specified topic does not found
    [✖]  code: 404 reason: Topic not found

    #the topic name is not in the format of <tenant>/<namespace>/<topic> or <topic>
    [✖]  Invalid short topic name '<topic-name>', it should be in the format of <tenant>/<namespace>/<topic> or <topic>

    #the topic name is not in the format of <domain>://<tenant>/<namespace>/<topic>
    [✖]  Invalid complete topic name '<topic-name>', it should be in the format of <domain>://<tenant>/<namespace>/<topic>

    #the topic name is not in the format of <tenant>/<namespace>/<topic>
    [✖]  Invalid topic name '<topic-name>', it should be in the format of<tenant>/<namespace>/<topic>

    #the namespace name is not in the format of <tenant>/<namespace>
    [✖]  The complete name of namespace is invalid. complete name : <namespace-complete-name>

    #the tenant name and(or) namespace name is empty
    [✖]  Invalid tenant or namespace. [<tenant>/<namespace>]

    #the tenant name contains unsupported special chars. the alphanumeric (a-zA-Z0-9) and the special chars (-=:.%)  is allowed
    [✖]  Tenant name include unsupported special chars. tenant : [<namespace>]

    #the namespace name contains unsupported special chars. the  alphanumeric (a-zA-Z0-9) and the special chars (-=:.%) is allowed
    [✖]  Namespace name include unsupported special chars. namespace : [<namespace>]

Usage: pulsarctl topics offload [flags]

offload-status

➜  pulsarctl git:(topic-operation) ./pulsarctl topic offload-status -h
USED FOR:
    This command is used for checking the status of data offloading from a persistent topic to long-term storage.

REQUIRED PERMISSION:
    This command requires tenant admin permissions.

EXAMPLES:
    #Check the status of data offloading from a topic <persistent-topic-name> to long-term storage
    pulsarctl topic offload-status <persistent-topic-name>

    #Wait for offloading to complete
    pulsarctl topic offload-status --wait <persistent-topic-name>

OUTPUT:
    #normal output
    Offload was a success

    #Offloading is not running
    Offload has not been run for <topic-name> since broker startup

    #Offloading is running
    Offload is currently running

    #Offload is error
    Error in offload

    #the topic name is not specified
    [✖]  only one argument is allowed to be used as a name

    #the specified topic does not found
    [✖]  code: 404 reason: Topic not found

    #the topic name is not in the format of <tenant>/<namespace>/<topic> or <topic>
    [✖]  Invalid short topic name '<topic-name>', it should be in the format of <tenant>/<namespace>/<topic> or <topic>

    #the topic name is not in the format of <domain>://<tenant>/<namespace>/<topic>
    [✖]  Invalid complete topic name '<topic-name>', it should be in the format of <domain>://<tenant>/<namespace>/<topic>

    #the topic name is not in the format of <tenant>/<namespace>/<topic>
    [✖]  Invalid topic name '<topic-name>', it should be in the format of<tenant>/<namespace>/<topic>

    #the namespace name is not in the format of <tenant>/<namespace>
    [✖]  The complete name of namespace is invalid. complete name : <namespace-complete-name>

    #the tenant name and(or) namespace name is empty
    [✖]  Invalid tenant or namespace. [<tenant>/<namespace>]

    #the tenant name contains unsupported special chars. the alphanumeric (a-zA-Z0-9) and the special chars (-=:.%)  is allowed
    [✖]  Tenant name include unsupported special chars. tenant : [<namespace>]

    #the namespace name contains unsupported special chars. the  alphanumeric (a-zA-Z0-9) and the special chars (-=:.%) is allowed
    [✖]  Namespace name include unsupported special chars. namespace : [<namespace>]

Usage: pulsarctl topics offload-status [flags]

unload

➜  pulsarctl git:(topic-operation) ./pulsarctl topic unload -h
USED FOR:
    This command is used for unloading a topic.

REQUIRED PERMISSION:
    This command requires super-user permissions.

EXAMPLES:
    #Unload a topic <topic-name>
    pulsarctl topic unload <topic-name>

OUTPUT:
    #normal output
    Unload topic <topic-name> successfully

    #the topic name is not specified
    [✖]  only one argument is allowed to be used as a name

    #the specified topic does not found
    [✖]  code: 404 reason: Topic not found

    #the topic name is not in the format of <tenant>/<namespace>/<topic> or <topic>
    [✖]  Invalid short topic name '<topic-name>', it should be in the format of <tenant>/<namespace>/<topic> or <topic>

    #the topic name is not in the format of <domain>://<tenant>/<namespace>/<topic>
    [✖]  Invalid complete topic name '<topic-name>', it should be in the format of <domain>://<tenant>/<namespace>/<topic>

    #the topic name is not in the format of <tenant>/<namespace>/<topic>
    [✖]  Invalid topic name '<topic-name>', it should be in the format of<tenant>/<namespace>/<topic>

    #the namespace name is not in the format of <tenant>/<namespace>
    [✖]  The complete name of namespace is invalid. complete name : <namespace-complete-name>

    #the tenant name and(or) namespace name is empty
    [✖]  Invalid tenant or namespace. [<tenant>/<namespace>]

    #the tenant name contains unsupported special chars. the alphanumeric (a-zA-Z0-9) and the special chars (-=:.%)  is allowed
    [✖]  Tenant name include unsupported special chars. tenant : [<namespace>]

    #the namespace name contains unsupported special chars. the  alphanumeric (a-zA-Z0-9) and the special chars (-=:.%) is allowed
    [✖]  Namespace name include unsupported special chars. namespace : [<namespace>]

Usage: pulsarctl topics unload [flags]

terminate

➜  pulsarctl git:(topic-operation) ./pulsarctl topic terminate -h
USED FOR:
    This command is used for terminating a topic and don't allow any more messages to  be published.

REQUIRED PERMISSION:
    This command requires tenant admin permissions.

EXAMPLES:
    #Terminate a topic <topic-name> and don't allow any messages to be published
    pulsarctl topic terminate <topic-name>

OUTPUT:
    #normal output
    Topic <topic-name> successfully terminated at <message-id>

    #the topic name is not specified
    [✖]  only one argument is allowed to be used as a name

    #the topic name is not in the format of <tenant>/<namespace>/<topic> or <topic>
    [✖]  Invalid short topic name '<topic-name>', it should be in the format of <tenant>/<namespace>/<topic> or <topic>

    #the topic name is not in the format of <domain>://<tenant>/<namespace>/<topic>
    [✖]  Invalid complete topic name '<topic-name>', it should be in the format of <domain>://<tenant>/<namespace>/<topic>

    #the topic name is not in the format of <tenant>/<namespace>/<topic>
    [✖]  Invalid topic name '<topic-name>', it should be in the format of<tenant>/<namespace>/<topic>

    #the namespace name is not in the format of <tenant>/<namespace>
    [✖]  The complete name of namespace is invalid. complete name : <namespace-complete-name>

    #the tenant name and(or) namespace name is empty
    [✖]  Invalid tenant or namespace. [<tenant>/<namespace>]

    #the tenant name contains unsupported special chars. the alphanumeric (a-zA-Z0-9) and the special chars (-=:.%)  is allowed
    [✖]  Tenant name include unsupported special chars. tenant : [<namespace>]

    #the namespace name contains unsupported special chars. the  alphanumeric (a-zA-Z0-9) and the special chars (-=:.%) is allowed
    [✖]  Namespace name include unsupported special chars. namespace : [<namespace>]

Usage: pulsarctl topics terminate [flags]

sijie

This pull request contains too many unrelated groups of commands. I would suggest breaking it down into multiple pull requests.

offload
terminate
compaction
status

sijie · 2019-09-17T12:37:15Z

-			"so please make sure the service urls provided in this command are reachable\n" +
-			"between clusters.\n" +
-			"\n" +
+		"This command is used for adding the configuration data for a cluster.\n"+


I see you are change formats in different pull requests. sometimes you add spaces and sometimes delete the spaces. I would suggest avoiding including style or code-convention changes in a huge pull request. if we need enforce a certain code convention, please enforce in the CI. /cc @wolfstudy

sijie · 2019-09-17T12:38:14Z

 func CheckTopicNameTwoArgs(args []string) error {
 	if len(args) != 2 {
-		return errors.New("need to specified the topic name and the partitions")
+		return errors.New("only two argument is allowed to be used as names")


why do you change the message here? If this method is used for other purpose, please create a new method and make sure the error message is self-explained and specific.

sijie · 2019-09-17T12:39:18Z

+	admin := cmdutils.NewPulsarClient()
+	err = admin.Topics().Compact(*topic)
+	if err == nil {
+		vc.Command.Printf("Sending compact topic %s request successfully/n", topic.String())


Suggested change

vc.Command.Printf("Sending compact topic %s request successfully/n", topic.String())

vc.Command.Printf("Started compacting topic %s successfully/n", topic.String())

sijie · 2019-09-17T12:39:48Z

+
+func CompactStatusCmd(vc *cmdutils.VerbCmd)  {
+	var desc LongDescription
+	desc.CommandUsedFor = "This command is used for getting status of compaction on a topic."


Suggested change

desc.CommandUsedFor = "This command is used for getting status of compaction on a topic."

desc.CommandUsedFor = "This command is used for getting the compaction status of a topic."

Please fix the example output as well.

sijie · 2019-09-17T12:41:15Z

+
+	var examples []Example
+	offload := Example{
+		Desc:    "Trigger offload the data from a topic <topic-name> to long-term storage and " +


Suggested change

Desc: "Trigger offload the data from a topic <topic-name> to long-term storage and " +

Desc: "Trigger offloading the data from a topic <topic-name> to a long-term storage and " +

Please also fix all the output examples.

sijie · 2019-09-17T12:43:01Z

+	var examples []Example
+	offload := Example{
+		Desc:    "Trigger offload the data from a topic <topic-name> to long-term storage and " +
+			"keep <threshold> size data in BookKeeper (e.g. 10M, 5G, default is byte)",


keep the configured amount of data in BookKeeper only

sijie · 2019-09-17T12:43:45Z

+	return nil
+}
+
+func validateSize(s string) (int64, error) {


Can you please find any existing libraries already did that?

sijie · 2019-09-17T12:45:31Z

+	}
+
+	errorOut := Output{
+		Desc: "Offload is error",


Please fix this sentence

sijie · 2019-09-17T12:45:51Z

+
+	notRun := Output{
+		Desc: "Offloading is not running",
+		Out: "Offload has not been run for <topic-name> since broker startup",


Please fix the syntax

sijie · 2019-09-17T12:46:28Z

+		}
+	}
+
+	switch status.Status {


Please make sure all the output sentences include namespace name.

sijie · 2019-09-17T12:53:05Z

Also it seems that #56 contains changes from #46. so which one I should review?

zymap · 2019-09-24T09:59:50Z

This pull request contains too many unrelated groups of commands. I would suggest breaking it down into multiple pull requests.

offload

terminate

compaction

status

@sijie This pull request breaks down as #73 #74 #75 #76. PTAL

zymap added 18 commits September 11, 2019 20:44

Add partitioned topic command CURD

9362444

Revert changes

bcba093

Add test for command partitioned topic

a2ae20f

Revert the refactor

6510c8d

Add non-persistent topic commands

b32bf3b

Revert changes

59c3f58

Revert changes

4a12631

Fix typo

b0a9dee

Add partitions value check

b89037f

Move to a new package

823c80f

Add topic stats command

12b1205

Rename partitioned-stats to partition-stats

cea7aa8

Combine partitioned topic and non-partitioned topic stats command

8a430f4

Remove partitioned stats file

1538806

Fix descriptions

2adb8ae

Fix conflict

65e2e87

Remove unused code

a839fac

Add topic operation commands

28d9248

zymap added the area/pulsar/topics label Sep 13, 2019

zymap requested a review from sijie September 13, 2019 16:11

zymap self-assigned this Sep 13, 2019

zymap removed the request for review from sijie September 13, 2019 16:14

zymap added 2 commits September 14, 2019 00:20

Revert changes

01f7829

Add topic command terminate

ebebd2d

sijie requested changes Sep 17, 2019

View reviewed changes

sijie mentioned this pull request Sep 18, 2019

pulsarctl MVP #2

Closed

29 tasks

zymap added 2 commits September 23, 2019 16:05

Merge branch 'master' into topic-operation

6e907bf

Merge branch 'master' into topic-operation

64ea9ac

zymap closed this Oct 10, 2019

wolfstudy deleted the topic-operation branch October 11, 2019 06:26

	vc.Command.Printf("Sending compact topic %s request successfully/n", topic.String())
	vc.Command.Printf("Started compacting topic %s successfully/n", topic.String())

	desc.CommandUsedFor = "This command is used for getting status of compaction on a topic."
	desc.CommandUsedFor = "This command is used for getting the compaction status of a topic."

	Desc: "Trigger offload the data from a topic <topic-name> to long-term storage and " +
	Desc: "Trigger offloading the data from a topic <topic-name> to a long-term storage and " +

Uh oh!

Conversation

zymap commented Sep 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sijie left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sijie commented Sep 17, 2019

Uh oh!

zymap commented Sep 24, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zymap commented Sep 13, 2019 •

edited

Loading