[SPARK-57451][CORE] Remove the check rejecting spark.authenticate.secret with Master REST server

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR removes the standalone Master check-code that rejects spark.authenticate.secret when the Master REST server (spark.master.rest.enabled) is enabled.

spark/core/src/main/scala/org/apache/spark/deploy/master/Master.scala

Lines 138 to 144 in 088071d

	{
	val authKey = SecurityManager.SPARK_AUTH_SECRET_CONF
	require(conf.getOption(authKey).isEmpty || !restServerEnabled,
	s"The RestSubmissionServer does not support authentication via ${authKey}. Either turn " +
	"off the RestSubmissionServer with spark.master.rest.enabled=false, or do not use " +
	"authentication.")
	}

For the record, the check was introduced at Apache Spark 2.4.0. And, currently, it's outdated.

• [SPARK-25088][CORE][MESOS][DOCS] Update Rest Server docs & defaults. #22071

Why are the changes needed?

spark.authenticate.secret (the RPC authentication secret) and spark.master.rest.enabled (the standalone submission REST server) are independent concerns, but the removed check-code coupled them by failing Master startup whenever both were set.

Since Apache Spark 4.1.0, spark.master.rest.enabled defaults to true, this check-code forced any cluster using RPC authentication to disable the REST server. This is wrong. We don't need to block like this because the REST server is protected independently like the following.

• [SPARK-49103][CORE] Support spark.master.rest.filters #47595 (Apache Spark 4.0.0)
• [SPARK-49104][CORE][DOCS] Document JWSFilter usage in Spark UI and REST API and rename parameter to secretKey #47596
• [SPARK-51165][CORE] Enable spark.master.rest.enabled by default #49894 (Apache Spark 4.1.0)

Does this PR introduce any user-facing change?

Previously, starting a standalone Master with spark.authenticate.secret set and spark.master.rest.enabled=true (the default) failed with an IllegalArgumentException. After this PR, the Master starts normally with both configured securely.

Although this is a bug fix by enabling a previous-blocked code path. So, technically there is no loss from the user perspective.

How was this patch tested?

Added a unit test in MasterSuite that verifies a Master can be created with both spark.master.rest.enabled=true and spark.authenticate.secret set.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

[dongjoon-hyun]

Could you review this when you have some time, @peter-toth ?

[dongjoon-hyun]

Thank you so much, @peter-toth !

[dongjoon-hyun]

Merged to master/4.x for now because we have Apache Spark 4.2.0 RC3 vote currently.

• If RC3 fails, I will consider to cherry-pick this to branch-4.2 and older.
• If RC3 succeeds, this will be Apache Spark 4.2.1.

cc @huaxingao