[proxy] Fix proxy to be able to re-send request body - #5361
Merged
Conversation
addisonj
force-pushed
the
fix_proxy
branch
2 times, most recently
from
October 11, 2019 07:00
3ff54c8 to
3abfe71
Compare
Contributor
Author
|
Okay, confirmed this works... not sure of an easy way to unit test it. I also want to add more one more change here to fix the strack trace on redirects that is confusing |
Contributor
Author
|
rerun integration tests |
Contributor
I think it should be happening each time the proxy is handling a 307 and the request has a body. I was forcing multiple GET request to the proxy. The idea is that the proxy will round-robin the requests on the available brokers. We could do the same for a PUT/POST request. |
Contributor
|
run java8 tests |
Contributor
Author
|
run java8 tests |
Fixes apache#5360 This adds a small cache of the request body to ensure that it can be re-sent. TODO: still needs tested
Contributor
Author
|
rerun integration tests |
sijie
approved these changes
Oct 24, 2019
Member
|
run java8 tests |
merlimat
approved these changes
Oct 24, 2019
wolfstudy
pushed a commit
that referenced
this pull request
Nov 20, 2019
sijie
pushed a commit
that referenced
this pull request
Jun 22, 2021
Fixes #10908 ### Motivation Pulsar Proxy uses a lot of heap memory when uploading large function jar files. This also leads to high GC activity since a continuous block of memory (byte array for the size of the upload) is allocated. GC will have to do compaction for the heap (which gets fragmented) to find a continuous block of memory. This is the reason why allocating large arrays are costly from GC perspective. The buffering solution added as part of #5361. The solution buffers also very large uploads to memory. ### Modifications * Limit the replay buffer size to a configurable limit which defaults to 5MB. This is configured with the `httpInputMaxReplayBufferSize` proxy configuration parameter. * Add unit test to see that buffer size gets limited * Add unit test for #5361
eolivelli
pushed a commit
to datastax/pulsar
that referenced
this pull request
Jun 23, 2021
Fixes apache#10908 ### Motivation Pulsar Proxy uses a lot of heap memory when uploading large function jar files. This also leads to high GC activity since a continuous block of memory (byte array for the size of the upload) is allocated. GC will have to do compaction for the heap (which gets fragmented) to find a continuous block of memory. This is the reason why allocating large arrays are costly from GC perspective. The buffering solution added as part of apache#5361. The solution buffers also very large uploads to memory. ### Modifications * Limit the replay buffer size to a configurable limit which defaults to 5MB. This is configured with the `httpInputMaxReplayBufferSize` proxy configuration parameter. * Add unit test to see that buffer size gets limited * Add unit test for apache#5361 (cherry picked from commit 2324618)
yangl
pushed a commit
to yangl/pulsar
that referenced
this pull request
Jun 23, 2021
Fixes apache#10908 ### Motivation Pulsar Proxy uses a lot of heap memory when uploading large function jar files. This also leads to high GC activity since a continuous block of memory (byte array for the size of the upload) is allocated. GC will have to do compaction for the heap (which gets fragmented) to find a continuous block of memory. This is the reason why allocating large arrays are costly from GC perspective. The buffering solution added as part of apache#5361. The solution buffers also very large uploads to memory. ### Modifications * Limit the replay buffer size to a configurable limit which defaults to 5MB. This is configured with the `httpInputMaxReplayBufferSize` proxy configuration parameter. * Add unit test to see that buffer size gets limited * Add unit test for apache#5361
kaushik-develop
pushed a commit
to kaushik-develop/pulsar
that referenced
this pull request
Jun 24, 2021
Fixes apache#10908 ### Motivation Pulsar Proxy uses a lot of heap memory when uploading large function jar files. This also leads to high GC activity since a continuous block of memory (byte array for the size of the upload) is allocated. GC will have to do compaction for the heap (which gets fragmented) to find a continuous block of memory. This is the reason why allocating large arrays are costly from GC perspective. The buffering solution added as part of apache#5361. The solution buffers also very large uploads to memory. ### Modifications * Limit the replay buffer size to a configurable limit which defaults to 5MB. This is configured with the `httpInputMaxReplayBufferSize` proxy configuration parameter. * Add unit test to see that buffer size gets limited * Add unit test for apache#5361
nicoloboschi
pushed a commit
to datastax/pulsar
that referenced
this pull request
Feb 28, 2022
Fixes apache#10908 Pulsar Proxy uses a lot of heap memory when uploading large function jar files. This also leads to high GC activity since a continuous block of memory (byte array for the size of the upload) is allocated. GC will have to do compaction for the heap (which gets fragmented) to find a continuous block of memory. This is the reason why allocating large arrays are costly from GC perspective. The buffering solution added as part of apache#5361. The solution buffers also very large uploads to memory. * Limit the replay buffer size to a configurable limit which defaults to 5MB. This is configured with the `httpInputMaxReplayBufferSize` proxy configuration parameter. * Add unit test to see that buffer size gets limited * Add unit test for apache#5361 (cherry picked from commit 2324618) (cherry picked from commit 7fa88cc)
bharanic-dev
pushed a commit
to bharanic-dev/pulsar
that referenced
this pull request
Mar 18, 2022
Fixes apache#10908 ### Motivation Pulsar Proxy uses a lot of heap memory when uploading large function jar files. This also leads to high GC activity since a continuous block of memory (byte array for the size of the upload) is allocated. GC will have to do compaction for the heap (which gets fragmented) to find a continuous block of memory. This is the reason why allocating large arrays are costly from GC perspective. The buffering solution added as part of apache#5361. The solution buffers also very large uploads to memory. ### Modifications * Limit the replay buffer size to a configurable limit which defaults to 5MB. This is configured with the `httpInputMaxReplayBufferSize` proxy configuration parameter. * Add unit test to see that buffer size gets limited * Add unit test for apache#5361
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5360
This adds a small cache of the request body to ensure that it can be
re-sent.