Modify authz tests of GET_TOPICS and PACKAGES op by yuruguo · Pull Request #12771 · apache/pulsar

yuruguo · 2021-11-12T07:23:42Z

Motivation

Integration authz test of GetTopicsOfNamespaceWithAuthTest.java and PackagesOpsWithAuthTest.java is too much heavyweight for check GET_TOPICS and PACKAGES namespace op.

In fact, we can add a small amount of code in AuthorizationProducerConsumerTest.java to replace with above integration test.

Documentation

no-need-doc

eolivelli

LGTM

RocMarshal

Good job. @yuruguo Thanks for the contribution. From my perspective this looks good. I also like to get concise in coding.

RobertIndie

I think we should remove the test class here:

pulsar/tests/integration/src/test/resources/pulsar-auth.xml

Line 25 in c3da145

    
           <class name="org.apache.pulsar.tests.integration.auth.admin.PackagesOpsWithAuthTest"/>

### Motivation Integration authz test of [GetTopicsOfNamespaceWithAuthTest.java](https://github.com/apache/pulsar/blob/master/tests/integration/src/test/java/org/apache/pulsar/tests/integration/auth/admin/GetTopicsOfNamespaceWithAuthTest.java) and [PackagesOpsWithAuthTest.java](https://github.com/apache/pulsar/blob/master/tests/integration/src/test/java/org/apache/pulsar/tests/integration/auth/admin/PackagesOpsWithAuthTest.java) is too much heavyweight for check GET_TOPICS and PACKAGES namespace op. In fact, we can add a small amount of code in [AuthorizationProducerConsumerTest.java](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/test/java/org/apache/pulsar/client/api/AuthorizationProducerConsumerTest.java) to replace with above integration test.

…Role (#19557) ### Motivation I broke the Pulsar Proxy with #19455 because that PR requires that when `X-Original-Principal` is supplied, the auth role must be a proxy role. This is not always the case for proxied admin requests. This PR seeks to fix that incorrect assumption by changing the way verification is done for the roles. Specifically, when the two roles are the same and they are not a proxy role, we will consider it a valid combination. Note that there is no inefficiency in this solution because When the `authenticatedPrincipal` is not a proxy role, that is the only role that is authenticated. Note also that we do not let the binary protocol authenticate this way, and that is consistent with the way the pulsar proxy forwards authentication data. Currently, we do the following when authentication is enabled in the proxy: 1. Authenticate the client's http request and put the resulting role in the `X-Original-Principal` header for the call to the broker. https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L370-L373 2. Copy the `Authorization` header into the broker's http request: https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L232-L236 3. Configure the proxy's http client to use client TLS authentication (when configured): https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L269-L277 The problem with #19455 is that it assumes the proxy supplies its own authentication data. However, that only happens when using TLS authentication. Otherwise, the proxy forwards the client's authentication data in the `Authorization` header. As such, calls will fail because the `X-Original-Principal` header supplied without using a proxy role. ### Modifications * Consider the `authenticatedPrincipal` and the `originalPrincipal` a valid pair when they are equal and are not a `proxyRole` for http requests. ### Alternative Solutions I initially proposed that we only add the `X-Original-Principal` when we are using the proxy's authentication (see the first commit). I decided this solution is not ideal because it doesn't solve the problem, it doesn't make the brokers backwards compatible, and there isn't actually any inefficiency in passing the role as a header. ### Verifying this change When cherry-picking #19455 to branch-2.9, I discovered that `PackagesOpsWithAuthTest#testPackagesOps` was consistently failing because of the way the proxy supplies authentication data when proxying http requests. That test was removed by #12771, which explains why I didn't catch the error sooner. This PR includes a test that fails without this change. Note that the primary issue must be that we didn't have any tests doing authentication forwarding through the proxy. Now we will have both relevant tests where the proxy is and is not authenticating. ### Does this pull request potentially affect one of the following parts: This is not a breaking change. ### Documentation - [x] `doc-required` ### Matching PR in forked repository PR in forked repository: michaeljmarshall#31

…Role (#19557) I broke the Pulsar Proxy with #19455 because that PR requires that when `X-Original-Principal` is supplied, the auth role must be a proxy role. This is not always the case for proxied admin requests. This PR seeks to fix that incorrect assumption by changing the way verification is done for the roles. Specifically, when the two roles are the same and they are not a proxy role, we will consider it a valid combination. Note that there is no inefficiency in this solution because When the `authenticatedPrincipal` is not a proxy role, that is the only role that is authenticated. Note also that we do not let the binary protocol authenticate this way, and that is consistent with the way the pulsar proxy forwards authentication data. Currently, we do the following when authentication is enabled in the proxy: 1. Authenticate the client's http request and put the resulting role in the `X-Original-Principal` header for the call to the broker. https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L370-L373 2. Copy the `Authorization` header into the broker's http request: https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L232-L236 3. Configure the proxy's http client to use client TLS authentication (when configured): https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L269-L277 The problem with #19455 is that it assumes the proxy supplies its own authentication data. However, that only happens when using TLS authentication. Otherwise, the proxy forwards the client's authentication data in the `Authorization` header. As such, calls will fail because the `X-Original-Principal` header supplied without using a proxy role. * Consider the `authenticatedPrincipal` and the `originalPrincipal` a valid pair when they are equal and are not a `proxyRole` for http requests. I initially proposed that we only add the `X-Original-Principal` when we are using the proxy's authentication (see the first commit). I decided this solution is not ideal because it doesn't solve the problem, it doesn't make the brokers backwards compatible, and there isn't actually any inefficiency in passing the role as a header. When cherry-picking #19455 to branch-2.9, I discovered that `PackagesOpsWithAuthTest#testPackagesOps` was consistently failing because of the way the proxy supplies authentication data when proxying http requests. That test was removed by #12771, which explains why I didn't catch the error sooner. This PR includes a test that fails without this change. Note that the primary issue must be that we didn't have any tests doing authentication forwarding through the proxy. Now we will have both relevant tests where the proxy is and is not authenticating. This is not a breaking change. - [x] `doc-required` PR in forked repository: michaeljmarshall#31 (cherry picked from commit d4be954)

michaeljmarshall · 2023-02-22T21:33:50Z

Just want to mention that this test re-write removed test coverage for admin client to proxy to broker with different kinds of authentication. I fixed it here #19557.

…Role (#19557) I broke the Pulsar Proxy with #19455 because that PR requires that when `X-Original-Principal` is supplied, the auth role must be a proxy role. This is not always the case for proxied admin requests. This PR seeks to fix that incorrect assumption by changing the way verification is done for the roles. Specifically, when the two roles are the same and they are not a proxy role, we will consider it a valid combination. Note that there is no inefficiency in this solution because When the `authenticatedPrincipal` is not a proxy role, that is the only role that is authenticated. Note also that we do not let the binary protocol authenticate this way, and that is consistent with the way the pulsar proxy forwards authentication data. Currently, we do the following when authentication is enabled in the proxy: 1. Authenticate the client's http request and put the resulting role in the `X-Original-Principal` header for the call to the broker. https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L370-L373 2. Copy the `Authorization` header into the broker's http request: https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L232-L236 3. Configure the proxy's http client to use client TLS authentication (when configured): https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L269-L277 The problem with #19455 is that it assumes the proxy supplies its own authentication data. However, that only happens when using TLS authentication. Otherwise, the proxy forwards the client's authentication data in the `Authorization` header. As such, calls will fail because the `X-Original-Principal` header supplied without using a proxy role. * Consider the `authenticatedPrincipal` and the `originalPrincipal` a valid pair when they are equal and are not a `proxyRole` for http requests. I initially proposed that we only add the `X-Original-Principal` when we are using the proxy's authentication (see the first commit). I decided this solution is not ideal because it doesn't solve the problem, it doesn't make the brokers backwards compatible, and there isn't actually any inefficiency in passing the role as a header. When cherry-picking #19455 to branch-2.9, I discovered that `PackagesOpsWithAuthTest#testPackagesOps` was consistently failing because of the way the proxy supplies authentication data when proxying http requests. That test was removed by #12771, which explains why I didn't catch the error sooner. This PR includes a test that fails without this change. Note that the primary issue must be that we didn't have any tests doing authentication forwarding through the proxy. Now we will have both relevant tests where the proxy is and is not authenticating. This is not a breaking change. - [x] `doc-required` PR in forked repository: michaeljmarshall#31 (cherry picked from commit d4be954) (cherry picked from commit 5f5551d)

…Role (apache#19557) I broke the Pulsar Proxy with apache#19455 because that PR requires that when `X-Original-Principal` is supplied, the auth role must be a proxy role. This is not always the case for proxied admin requests. This PR seeks to fix that incorrect assumption by changing the way verification is done for the roles. Specifically, when the two roles are the same and they are not a proxy role, we will consider it a valid combination. Note that there is no inefficiency in this solution because When the `authenticatedPrincipal` is not a proxy role, that is the only role that is authenticated. Note also that we do not let the binary protocol authenticate this way, and that is consistent with the way the pulsar proxy forwards authentication data. Currently, we do the following when authentication is enabled in the proxy: 1. Authenticate the client's http request and put the resulting role in the `X-Original-Principal` header for the call to the broker. https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L370-L373 2. Copy the `Authorization` header into the broker's http request: https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L232-L236 3. Configure the proxy's http client to use client TLS authentication (when configured): https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L269-L277 The problem with apache#19455 is that it assumes the proxy supplies its own authentication data. However, that only happens when using TLS authentication. Otherwise, the proxy forwards the client's authentication data in the `Authorization` header. As such, calls will fail because the `X-Original-Principal` header supplied without using a proxy role. * Consider the `authenticatedPrincipal` and the `originalPrincipal` a valid pair when they are equal and are not a `proxyRole` for http requests. I initially proposed that we only add the `X-Original-Principal` when we are using the proxy's authentication (see the first commit). I decided this solution is not ideal because it doesn't solve the problem, it doesn't make the brokers backwards compatible, and there isn't actually any inefficiency in passing the role as a header. When cherry-picking apache#19455 to branch-2.9, I discovered that `PackagesOpsWithAuthTest#testPackagesOps` was consistently failing because of the way the proxy supplies authentication data when proxying http requests. That test was removed by apache#12771, which explains why I didn't catch the error sooner. This PR includes a test that fails without this change. Note that the primary issue must be that we didn't have any tests doing authentication forwarding through the proxy. Now we will have both relevant tests where the proxy is and is not authenticating. This is not a breaking change. - [x] `doc-required` PR in forked repository: #31 (cherry picked from commit d4be954) (cherry picked from commit 5f5551d) (cherry picked from commit 4da2487)

…Role (apache#19557) I broke the Pulsar Proxy with apache#19455 because that PR requires that when `X-Original-Principal` is supplied, the auth role must be a proxy role. This is not always the case for proxied admin requests. This PR seeks to fix that incorrect assumption by changing the way verification is done for the roles. Specifically, when the two roles are the same and they are not a proxy role, we will consider it a valid combination. Note that there is no inefficiency in this solution because When the `authenticatedPrincipal` is not a proxy role, that is the only role that is authenticated. Note also that we do not let the binary protocol authenticate this way, and that is consistent with the way the pulsar proxy forwards authentication data. Currently, we do the following when authentication is enabled in the proxy: 1. Authenticate the client's http request and put the resulting role in the `X-Original-Principal` header for the call to the broker. https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L370-L373 2. Copy the `Authorization` header into the broker's http request: https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L232-L236 3. Configure the proxy's http client to use client TLS authentication (when configured): https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L269-L277 The problem with apache#19455 is that it assumes the proxy supplies its own authentication data. However, that only happens when using TLS authentication. Otherwise, the proxy forwards the client's authentication data in the `Authorization` header. As such, calls will fail because the `X-Original-Principal` header supplied without using a proxy role. * Consider the `authenticatedPrincipal` and the `originalPrincipal` a valid pair when they are equal and are not a `proxyRole` for http requests. I initially proposed that we only add the `X-Original-Principal` when we are using the proxy's authentication (see the first commit). I decided this solution is not ideal because it doesn't solve the problem, it doesn't make the brokers backwards compatible, and there isn't actually any inefficiency in passing the role as a header. When cherry-picking apache#19455 to branch-2.9, I discovered that `PackagesOpsWithAuthTest#testPackagesOps` was consistently failing because of the way the proxy supplies authentication data when proxying http requests. That test was removed by apache#12771, which explains why I didn't catch the error sooner. This PR includes a test that fails without this change. Note that the primary issue must be that we didn't have any tests doing authentication forwarding through the proxy. Now we will have both relevant tests where the proxy is and is not authenticating. This is not a breaking change. - [x] `doc-required` PR in forked repository: michaeljmarshall#31 (cherry picked from commit d4be954) (cherry picked from commit 5f5551d) (cherry picked from commit 4da2487)

…Role (apache#19557) I broke the Pulsar Proxy with apache#19455 because that PR requires that when `X-Original-Principal` is supplied, the auth role must be a proxy role. This is not always the case for proxied admin requests. This PR seeks to fix that incorrect assumption by changing the way verification is done for the roles. Specifically, when the two roles are the same and they are not a proxy role, we will consider it a valid combination. Note that there is no inefficiency in this solution because When the `authenticatedPrincipal` is not a proxy role, that is the only role that is authenticated. Note also that we do not let the binary protocol authenticate this way, and that is consistent with the way the pulsar proxy forwards authentication data. Currently, we do the following when authentication is enabled in the proxy: 1. Authenticate the client's http request and put the resulting role in the `X-Original-Principal` header for the call to the broker. https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L370-L373 2. Copy the `Authorization` header into the broker's http request: https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L232-L236 3. Configure the proxy's http client to use client TLS authentication (when configured): https://github.com/apache/pulsar/blob/38555851359f9cfc172650c387a58c5a03809e97/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L269-L277 The problem with apache#19455 is that it assumes the proxy supplies its own authentication data. However, that only happens when using TLS authentication. Otherwise, the proxy forwards the client's authentication data in the `Authorization` header. As such, calls will fail because the `X-Original-Principal` header supplied without using a proxy role. * Consider the `authenticatedPrincipal` and the `originalPrincipal` a valid pair when they are equal and are not a `proxyRole` for http requests. I initially proposed that we only add the `X-Original-Principal` when we are using the proxy's authentication (see the first commit). I decided this solution is not ideal because it doesn't solve the problem, it doesn't make the brokers backwards compatible, and there isn't actually any inefficiency in passing the role as a header. When cherry-picking apache#19455 to branch-2.9, I discovered that `PackagesOpsWithAuthTest#testPackagesOps` was consistently failing because of the way the proxy supplies authentication data when proxying http requests. That test was removed by apache#12771, which explains why I didn't catch the error sooner. This PR includes a test that fails without this change. Note that the primary issue must be that we didn't have any tests doing authentication forwarding through the proxy. Now we will have both relevant tests where the proxy is and is not authenticating. This is not a breaking change. - [x] `doc-required` PR in forked repository: #31 (cherry picked from commit d4be954) (cherry picked from commit 5f5551d) (cherry picked from commit 4da2487) (cherry picked from commit dc09681)

github-actions Bot assigned yuruguo Nov 12, 2021

github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Nov 12, 2021

eolivelli approved these changes Nov 12, 2021

View reviewed changes

RocMarshal approved these changes Nov 12, 2021

View reviewed changes

RobertIndie approved these changes Nov 12, 2021

View reviewed changes

RobertIndie reviewed Nov 12, 2021

View reviewed changes

Modify authz tests of GET_TOPICS and PACKAGES op

21c494c

codelipenghui approved these changes Nov 15, 2021

View reviewed changes

codelipenghui added this to the 2.10.0 milestone Nov 15, 2021

codelipenghui merged commit 22f2270 into apache:master Nov 15, 2021

michaeljmarshall mentioned this pull request Feb 17, 2023

[fix][broker] Allow proxy to pass same role for authRole and originalRole #19557

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Modify authz tests of GET_TOPICS and PACKAGES op - #12771

Modify authz tests of GET_TOPICS and PACKAGES op#12771
codelipenghui merged 1 commit into
apache:masterfrom
yuruguo:modify-authz-tests

yuruguo commented Nov 12, 2021 •

edited

Loading

Uh oh!

eolivelli left a comment

Uh oh!

RocMarshal left a comment

Uh oh!

RobertIndie left a comment

Uh oh!

michaeljmarshall commented Feb 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

yuruguo commented Nov 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Documentation

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

RocMarshal left a comment

Choose a reason for hiding this comment

Uh oh!

RobertIndie left a comment

Choose a reason for hiding this comment

Uh oh!

michaeljmarshall commented Feb 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

yuruguo commented Nov 12, 2021 •

edited

Loading