Proxy doesn't send request body on redirects

[addisonj]

Describe the bug
When using multiple brokers behind a proxy, requests with a request body from the client fail. See #3702 as an example. The following seems to be what happens:

• The client makes a request to proxy
• The proxy forward the requests to the broker
• The broker is not authoritative for that request, so it responds with a redirect
• The proxy follows the redirect but DOES NOT forward on the original request body
• The second broker, seeing a content-length header but never getting a body, waits for a full response but then times out
• The proxy responds with a 504 gateway timeout

To Reproduce
Steps to reproduce the behavior:

1. Run multiple brokers
2. Run a proxy in front of the brokers
3. make an admin request that takes a request body and must be routed to a specific broker, like creating a subscription to a specific ledger id (a default subscription doesn't send a request body)
4. Try a few times, you should eventually get a gateway timeout

This appears to be the case with and without TLS. I haven't yet tried it without authentication

Expected behavior
The request should succeed, with the request body being sent with the redirect

Screenshots
I validate this with tcp dumps:

The initial request to the first broker:

PUT /admin/v2/persistent/code/gerrit-events/message_bus%3Aref-updated/subscription/flink-pulsar-2a7cc468-6712-44b9-8b8f-0c8b86b78dae HTTP/1.1
Authorization: Bearer ...
User-Agent: Pulsar-Java-v2.4.1
Host: pulsar-pdx.bus-beta.insk8s.net:8443
Accept: application/json
Content-Type: application/json
Via: http/1.1 pulsar-beta-proxy-6b87d7f8cc-thmmp
X-Forwarded-For: 10.11.62.69
X-Forwarded-Proto: https
X-Forwarded-Host: pulsar-pdx.bus-beta.insk8s.net:8443
X-Forwarded-Server: 10.11.61.62
X-Original-Principal: code-admin
Content-Length: 82

{"ledgerId":9223372036854775807,"entryId":9223372036854775807,"partitionIndex":-1}

HTTP/1.1 307 Temporary Redirect
Date: Thu, 10 Oct 2019 23:11:41 GMT
Location: http://10.11.41.134:8080/admin/v2/persistent/code/gerrit-events/message_bus%3Aref-updated/subscription/flink-pulsar-2a7cc468-6712-44b9-8b8f-0c8b86b78dae?authoritative=false
broker-address: 10.11.54.198
Content-Length: 0
Server: Jetty(9.4.12.v20180830)

The redirect to the second broker (from the proxy)

PUT /admin/v2/persistent/code/gerrit-events/message_bus%3Aref-updated/subscription/flink-pulsar-2a7cc468-6712-44b9-8b8f-0c8b86b78dae?authoritative=false HTTP/1.1
User-Agent: Jetty/9.4.12.v20180830
User-Agent: Pulsar-Java-v2.4.1
Accept: application/json
Content-Type: application/json
Via: http/1.1 pulsar-beta-proxy-6b87d7f8cc-thmmp
X-Forwarded-For: 10.11.62.69
X-Forwarded-Proto: https
X-Forwarded-Host: pulsar-pdx.bus-beta.insk8s.net:8443
X-Forwarded-Server: 10.11.61.62
X-Original-Principal: code-admin
Authorization: Bearer ...
Host: 10.11.41.134:8080
Content-Length: 82

HTTP/1.1 500 Request failed.
Date: Thu, 10 Oct 2019 23:11:41 GMT
Content-Length: 0
Server: Jetty(9.4.12.v20180830)

Note that the content-length is set but no body is sent.

Desktop (please complete the following information):
Occurs with Pulsar 2.4.1 on linux

Additional context
I have spent a fair amount of time trying to debug this, here are some known details:

In AdminProxyHandler, we copy the request, see https://github.com/apache/pulsar/blob/v2.4.1/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L152

The super method here, https://github.com/eclipse/jetty.project/blob/jetty-9.4.12.v20180830/jetty-client/src/main/java/org/eclipse/jetty/client/HttpClient.java#L459, copies the content.

However, the contentProvider may be an iterator, that depending on the implementation, can't be reset. It isn't clear what implementation would be used by the proxy.

Additionally, this is made more difficult as it seems like the admin proxy isn't logging as expected. With debug level logs turned on, I didn't see logs from the proxy for some code paths I am sure it should be in.

I think the next step might be to have the AdminProxyHandler consume the request body before sending the first request and then re-attaching it for the first and any subsequent proxied requests

[addisonj]

Okay, one more follow up. It seems like the default ProxyServlet provides its own ContentProvider, https://github.com/eclipse/jetty.project/blob/jetty-9.4.12.v20180830/jetty-proxy/src/main/java/org/eclipse/jetty/proxy/ProxyServlet.java#L244, that is a really thing wrapper over this class: https://github.com/eclipse/jetty.project/blob/jetty-9.4.12.v20180830/jetty-client/src/main/java/org/eclipse/jetty/client/util/InputStreamContentProvider.java#L37.

As can be seen from the doc to the InputStreamContentProvider, the input stream can only be consumed once, additionally, no buffering is done by the proxied stream, that would seem to explain why the content can be replayed.

[merlimat]

Nice finding!

Seems similar to the fix for #4059. In that commit I had added a test to verify the auth headers were being propagated after a redirect, though I didn't think about testing with a request body.

[addisonj]

I have it implemented, going to to try it out and will push it up

[cdbartholomew]

This is definitely what is happening in #3702. I was just debugging that with tcpdump and ended up here. Great to see a fix is on its way. Thanks @addisonj and @merlimat.