[feat][io] AzureDataExplorer/Kusto Sink for Pulsar

[asaharn]

Motivation

This PR introduces an Azure Data Explorer (ADX) sink connector for Apache Pulsar. Streamlining the ingestion process, it solves the challenge of seamlessly transferring data from Pulsar to ADX clusters.

Modifications

We have added a new Azure Data Explorer (ADX) sink connector tailored for Apache Pulsar, enabling smooth and optimized data transfer from Pulsar to ADX clusters. The modification includes the integration of key functionalities and configurations necessary for seamless data ingestion.

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change added tests and can be verified as follows:

• Added end-2-end tests for the sink and can be tested by setting env vars

    export kustoAadAppId="" 
    export kustoAadAppSecret=""
    export kustoAadAuthorityID="tenentId"
    export kustoDatabase="<dbname>"
    export kustoCluster="https://ingest-<cluster-name>.kusto.windows.net"```
  
  

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: asaharn#2

[github-actions]

@asaharn Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

[lhotari]

Thanks for the contribution @asaharn ! Looks really good in general. A few small details to revisit.

[lhotari]

There's one more change needed for handling secrets. I'm sorry I missed that in the initial review.

[david-streamlio]

LGTM

[asaharn]

Thanks @david-streamlio.
@lhotari Made the changes for handling secrets.

[lhotari]

A few comments about .gitignore and the mockito-inline dependency that is no longer needed in Mockito 5.x.x.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 13.81215% with 156 lines in your changes missing coverage. Please review.

Project coverage is 73.58%. Comparing base (1b4127a) to head (6248a25).
Report is 1224 commits behind head on master.

Files with missing lines	Patch %	Lines
...rg/apache/pulsar/io/azuredataexplorer/ADXSink.java	0.00%	138 Missing ⚠️
...he/pulsar/io/azuredataexplorer/ADXPulsarEvent.java	0.00%	7 Missing ⚠️
...ache/pulsar/io/azuredataexplorer/ADXSinkUtils.java	0.00%	7 Missing ⚠️
...che/pulsar/io/azuredataexplorer/ADXSinkConfig.java	86.20%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #22006       +/-   ##
=============================================
+ Coverage     36.44%   73.58%   +37.14%     
- Complexity    12369    32572    +20203     
=============================================
  Files          1727     1874      +147     
  Lines        131879   139220     +7341     
  Branches      14419    15260      +841     
=============================================
+ Hits          48057   102445    +54388     
+ Misses        77404    28862    -48542     
- Partials       6418     7913     +1495     

Flag	Coverage Δ
inttests	24.67% <ø> (+0.48%)	⬆️
systests	24.36% <ø> (+0.46%)	⬆️
unittests	72.86% <13.81%> (+40.97%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...che/pulsar/io/azuredataexplorer/ADXSinkConfig.java	86.20% <86.20%> (ø)
...he/pulsar/io/azuredataexplorer/ADXPulsarEvent.java	0.00% <0.00%> (ø)
...ache/pulsar/io/azuredataexplorer/ADXSinkUtils.java	0.00% <0.00%> (ø)
...rg/apache/pulsar/io/azuredataexplorer/ADXSink.java	0.00% <0.00%> (ø)

... and 1447 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[lhotari]

Thanks for the great work @asaharn . Merging this.