[SPARK-43505][K8S] support env variables substitution and executor library path

[advancedxy]

What changes were proposed in this pull request?

1. add a new feature step to handle special environment variables, such as PATH=/usr/bin:$PATH
   • pull these special env variables and reorder them in a new config map
   • mount the config map in driver and/or executor's container
   • export these env variables by leverage sh/bash's . capability in the entrypoint.sh
   • update original container's env list with special envs excluded
2. support spark.executor.exraLibraryPath on K8S by leveraging the new feature step
3. update spark-rbac.yaml to including more permissions, which was found when add integration tests.

Why are the changes needed?

1. support spark.executor.extraLibraryPath on k8s
2. feature parity with spark on yarn. Spark on yarn handles this environment variables that need to be substituted correctly.
3. K8S itself cannot substitute variables in the container spec, we have to do some extra work at the entrypoint.sh

Does this PR introduce any user-facing change?

Yes. Before this PR, for spark running on K8S, spark.executor.extraLibraryPath doesn't take effect and cannot support
variable substitution.

How was this patch tested?

1. added new UTs
2. added new integration test
3. [WIP] verifing in the real prod cluster.

[advancedxy]

this doesn't work as expected as spark wouldn't create other resource when building executor pod.

Will fix and propose another solution.

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!