[SPARK-57430][CORE] Validate length before allocation in Encoders.ByteArrays.decode

[dongjoon-hyun]

What changes were proposed in this pull request?

Encoders.ByteArrays.decode() now validates the length prefix before allocating the array via Objects.checkFromIndexSize(0, length, buf.readableBytes()), which requires 0 <= length <= buf.readableBytes() and throws IndexOutOfBoundsException otherwise. After readInt(), the reader index is past the prefix, so readableBytes() equals the remaining payload bytes.

• Note that we use the Java built-in Objects.checkFromIndexSize because we cannot use Netty's checkReadableBytes. However, both methods throws IndexOutOfBoundsException identically for the error cases.
  • https://netty.io/4.2/api/io/netty/buffer/AbstractByteBuf.html#checkReadableBytes(int)
  • https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/Objects.html#checkFromIndexSize(int,int,int)

Why are the changes needed?

decode() allocated new byte[length] from an untrusted length.

• A negative value throws an opaque NegativeArraySizeException.
• An oversized value can trigger OutOfMemoryError on a corrupt or hostile frame.

Validating first fails fast with a clear error.

Note that OutOfMemoryError can happen even in a secure environment at the first parse on an unauthenticated channel.

spark/common/network-common/src/main/java/org/apache/spark/network/crypto/AuthRpcHandler.java

Lines 88 to 90 in b6450e6

	AuthMessage challenge;
	try {
	challenge = AuthMessage.decodeMessage(message);

spark/common/network-common/src/main/java/org/apache/spark/network/crypto/AuthMessage.java

Lines 53 to 64 in b6450e6

	public static AuthMessage decodeMessage(ByteBuffer buffer) {
	ByteBuf buf = Unpooled.wrappedBuffer(buffer);
	if (buf.readByte() != TAG_BYTE) {
	throw new IllegalArgumentException("Expected ClientChallenge, received something else.");
	}
	return new AuthMessage(
	Encoders.Strings.decode(buf), // AppID
	Encoders.ByteArrays.decode(buf), // Salt
	Encoders.ByteArrays.decode(buf)); // Ciphertext
	}

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs with newly added test cases in EncodersSuite.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

[dongjoon-hyun]

Thank you, @peter-toth .

[dongjoon-hyun]

Merged to master/4.x for now because we have Apache Spark 4.2.0 RC3 vote currently.

If it fails, I will consider to cherry-pick this to branch-4.2 and older.

cc @huaxingao