Skip to content

Add support for deduplication testing - #10246

Merged
merlimat merged 3 commits into
apache:masterfrom
klwilson227:SupportTestDeduplication
Apr 21, 2021
Merged

Add support for deduplication testing#10246
merlimat merged 3 commits into
apache:masterfrom
klwilson227:SupportTestDeduplication

Conversation

@klwilson227

Copy link
Copy Markdown
Contributor

Add some basic support for testing deduplication feature.

  1. Ability to set the producer name in the test client.
  2. Ability to set the sendTimeout in the test client.
  3. Ability to set the separator used between producername and threadnumber. As well as the topic and topic number. Defaults to '-'
  4. Ability to customize format each payload record if enabled.
    • Ability to customize/replace the formatting class which is used.
    • DefaultFormatter provided:
    • MetaData replacements: %p replaced with producer Name, %t nanoseconds when replacement occured, %i external messageId
    • Other replacements follow java sprintf formatting with %d %s %f %l with randomly generated strings or numbers.

@klwilson227
klwilson227 force-pushed the SupportTestDeduplication branch from 18831ac to f6d9802 Compare April 15, 2021 22:01

@merlimat merlimat left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the goal is testing the deduping mechanism, we should probably also add something in the performance consumer to validate that no messages with duplicated sequences are received.

Comment thread pulsar-testclient/pom.xml
@klwilson227
klwilson227 force-pushed the SupportTestDeduplication branch 2 times, most recently from d07e66f to bc293c8 Compare April 16, 2021 14:13
@klwilson227

Copy link
Copy Markdown
Contributor Author

Merlimat, RE the consumer side, I'm sure the consumer side would also benefit as well for a possible demo/software acceptance level validation. However, in my particular use case the validation and scanning for duplicates and lost records was easier implemented in the Data Sink which received the data hence I did not invest extra to build a data validation layer within pulsar test client, only the ability to publish data using both the dedup enabled and dedup disabled.

Since, I did not have the requirement, it was not coded. And I did not want to assume that this was a requirement, as a poor implementation may show both strengths and weaknesses in the platform. I can certainly take it in that direction. But would think that may be a separate PR.

@klwilson227
klwilson227 force-pushed the SupportTestDeduplication branch from bc293c8 to 636306a Compare April 16, 2021 14:31
@klwilson227
klwilson227 requested a review from merlimat April 20, 2021 15:21
@klwilson227

klwilson227 commented Apr 21, 2021

Copy link
Copy Markdown
Contributor Author

/pulsabot rerun CI-CPP,Python Tests

@merlimat merlimat added this to the 2.8.0 milestone Apr 21, 2021
@merlimat merlimat added the type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages label Apr 21, 2021
@merlimat
merlimat merged commit e4630a9 into apache:master Apr 21, 2021
@klwilson227
klwilson227 deleted the SupportTestDeduplication branch April 22, 2021 19:09
@lhotari

lhotari commented Apr 26, 2021

Copy link
Copy Markdown
Member

This PR introduced a flaky test, reported as #10378. I have created a PR to fix the flakiness in #10379 . Please review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/tool type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants