[fix][broker] Make authentication refresh threadsafe

[michaeljmarshall]

Fixes #19431

Motivation

The originalAuthState and the authState are not thread safe members of the ServerCnx class. As such, we should only access them on the context's event loop thread.

Modifications

• Replace scheduled task to check all connections for authentication refresh with a scheduled task per ServerCnx that will run on the appropriate event loop. This logic could theoretically be optimized to minimize scheduled tasks. However, I am following the keep alive implementation and making the assumption that the Netty scheduled tasks are not too expensive.
• Slightly refactor conditional logic in refreshAuthenticationCredentials method to remove the logic that it only short circuits when the ServerCnx is in Failed state. We should never call this method when the isActive flag has gone to false, and the only case when the state isn't Connected is when we recently failed due to an exception.

Verifying this change

Added new test and make sure existing tests pass. Also, I added some new assert lines. These only run when the correct JVM arg is passed.

Does this pull request potentially affect one of the following parts:

This is not a breaking change.

Documentation

• doc-not-needed

This is an internal improvement.

Matching PR in forked repository

PR in forked repository: michaeljmarshall#28

[michaeljmarshall]

/pulsarbot rerun-failure-checks

[codecov-commenter]

Codecov Report

❌ Patch coverage is 52.38095% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.27%. Comparing base (950ff44) to head (2d408cd).
⚠️ Report is 2843 commits behind head on master.

Files with missing lines	Patch %	Lines
...va/org/apache/pulsar/broker/service/ServerCnx.java	52.38%	13 Missing and 7 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19506      +/-   ##
============================================
- Coverage     63.73%   63.27%   -0.47%     
- Complexity    26234    26239       +5     
============================================
  Files          1843     1844       +1     
  Lines        135158   135238      +80     
  Branches      14856    14871      +15     
============================================
- Hits          86149    85575     -574     
- Misses        41176    41897     +721     
+ Partials       7833     7766      -67     

Flag	Coverage Δ
inttests	24.69% <0.00%> (+0.02%)	⬆️
systests	25.36% <21.42%> (+<0.01%)	⬆️
unittests	60.60% <52.38%> (-0.56%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...ulsar/broker/service/PulsarChannelInitializer.java	95.00% <ø> (+1.57%)	⬆️
...va/org/apache/pulsar/broker/service/ServerCnx.java	55.02% <52.38%> (-0.55%)	⬇️

... and 218 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[lhotari]

Great work @michaeljmarshall