HADOOP-19922. Add configurable warn threshold for large IPC RPC request data length by cxzl25 · Pull Request #8553 · apache/hadoop

cxzl25 · 2026-06-18T09:44:28Z

Description of PR

The IPC `Server` currently enforces a hard upper limit on RPC request size via `ipc.maximum.data.length` (default 128 MB). Requests that exceed this limit are rejected immediately. However, requests that are abnormally large but still within the limit pass through silently, potentially causing GC pressure and cascading failures before any operator is aware.

This patch adds a configurable warn threshold `ipc.data.length.warn.threshold` (default 64 MB). When a request exceeds this threshold but is still within the hard limit, a WARN log is emitted. Set to `-1` to disable.

How was this patch tested?

Added two unit tests to `TestIPC`:

`testWarnDataLengthThreshold`: verifies a WARN log is emitted when a request exceeds the configured threshold but is within the hard limit.
`testWarnDataLengthThresholdDisabled`: verifies no WARN log is emitted when the threshold is set to `-1` (disabled).

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?

AI Tooling

If an AI tool was used:

The PR includes the phrase "Contains content generated by "
where is the name of the AI tool used.
My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html

hadoop-yetus · 2026-06-18T13:44:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	47m 43s		trunk passed
+1 💚	compile	17m 18s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	compile	17m 59s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	checkstyle	1m 30s		trunk passed
+1 💚	mvnsite	1m 59s		trunk passed
+1 💚	javadoc	1m 25s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	1m 21s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	3m 14s		trunk passed
+1 💚	shadedclient	36m 52s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 15s		the patch passed
+1 💚	compile	16m 30s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javac	16m 30s		the patch passed
+1 💚	compile	18m 4s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	javac	18m 4s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 27s		the patch passed
+1 💚	mvnsite	1m 59s		the patch passed
+1 💚	javadoc	1m 23s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	1m 22s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	3m 22s		the patch passed
+1 💚	shadedclient	36m 49s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	24m 6s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 12s		The patch does not generate ASF License warnings.
		239m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8553/1/artifact/out/Dockerfile
GITHUB PR	#8553
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux e3c720f6ef2e 5.15.0-174-generic #184-Ubuntu SMP Fri Mar 13 18:41:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `0141ddd`
Default Java	Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8553/1/testReport/
Max. process+thread count	1273 (vs. ulimit of 10000)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8553/1/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

steveloughran

test?

Add two unit tests to TestIPC: - testWarnDataLengthThreshold: verifies a WARN is logged when a request exceeds ipc.data.length.warn.threshold but is within the hard limit - testWarnDataLengthThresholdDisabled: verifies no WARN is logged when the threshold is set to -1 (disabled) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

hadoop-yetus · 2026-06-23T08:46:43Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	47m 55s		trunk passed
+1 💚	compile	17m 58s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	compile	18m 6s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	checkstyle	1m 31s		trunk passed
+1 💚	mvnsite	2m 0s		trunk passed
+1 💚	javadoc	1m 25s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	1m 22s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	3m 14s		trunk passed
+1 💚	shadedclient	36m 50s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 15s		the patch passed
+1 💚	compile	16m 43s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javac	16m 43s		the patch passed
+1 💚	compile	17m 45s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	javac	17m 45s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 27s		the patch passed
+1 💚	mvnsite	1m 58s		the patch passed
+1 💚	javadoc	1m 22s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	1m 23s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	3m 26s		the patch passed
+1 💚	shadedclient	36m 41s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	24m 25s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 14s		The patch does not generate ASF License warnings.
		240m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8553/2/artifact/out/Dockerfile
GITHUB PR	#8553
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 7bc38b6999a2 5.15.0-174-generic #184-Ubuntu SMP Fri Mar 13 18:41:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ffdbab8`
Default Java	Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8553/2/testReport/
Max. process+thread count	3138 (vs. ulimit of 10000)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8553/2/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

HADOOP-19922

0141ddd

github-actions Bot added trunk Common labels Jun 18, 2026

steveloughran reviewed Jun 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-19922. Add configurable warn threshold for large IPC RPC request data length#8553

HADOOP-19922. Add configurable warn threshold for large IPC RPC request data length#8553
cxzl25 wants to merge 2 commits into
apache:trunkfrom
cxzl25:HADOOP-19922

cxzl25 commented Jun 18, 2026 •

edited

Loading

Uh oh!

hadoop-yetus commented Jun 18, 2026

Uh oh!

steveloughran left a comment

Uh oh!

hadoop-yetus commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

cxzl25 commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

AI Tooling

Uh oh!

hadoop-yetus commented Jun 18, 2026

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cxzl25 commented Jun 18, 2026 •

edited

Loading