Spark: Promote private SparkSubmitHook attributes used by strategy backends to public interface

[onlyarnav]

Description

Following the refactoring of SparkSubmitOperator into backend strategy classes (_KubernetesSparkSubmitBackend, _YarnSparkSubmitBackend, _StandaloneSparkSubmitBackend), several operator backend methods reach directly into private attributes and methods of SparkSubmitHook:

• hook._conf
• hook._kubernetes_driver_pod
• hook._yarn_application_id
• hook._poll_k8s_driver_via_api()

Reaching into private _ members across class boundaries weakens object encapsulation between SparkSubmitOperator and SparkSubmitHook.

Proposed Solution

Promote these key internal attributes and methods on SparkSubmitHook to clean public interface methods/properties:

1. Expose @property def conf(self) on SparkSubmitHook.
2. Expose @property def kubernetes_driver_pod(self) (getter and setter).
3. Expose @property def yarn_application_id(self).
4. Promote _poll_k8s_driver_via_api() to public poll_k8s_driver_via_api().

Update _SparkSubmitDeploymentBackend strategy classes in spark_submit.py to use these clean public hook methods.

Related Context

Follow-up thought from PR #68679 review.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct