[SPARK-47323][K8S] Support custom executor log urls#45464
Conversation
|
+CC @thejdeep |
There was a problem hiding this comment.
the external log service for K8s is likely to use namespace and pod name to query the logs, could you please expose NAMESPACE too?
There was a problem hiding this comment.
I have added the namespace.
There was a problem hiding this comment.
I think the Please check the documentation for your cluster manager to see which patterns are supported, if any. is sufficient, there is no need to list which manager supports this conf and which don't. That list easily gets out-dated.
EnricoMi
force-pushed
the
k8s-custom-executor-log-url
branch
from
80070ef to
2f896c8
Compare
|
@dongjoon-hyun What do you think? |
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
EnricoMi
force-pushed
the
k8s-custom-executor-log-url
branch
from
2f896c8 to
315a0bb
Compare
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Sorry for being late, @EnricoMi .
Apache Spark has been supported this feature. So, The configuration document is fixed in master/3.5/3.4. Could you try to follow the updated documentation?
spark.executorEnv.SPARK_EXECUTOR_ATTRIBUTE_APP_ID='$(SPARK_APPLICATION_ID)'
spark.executorEnv.SPARK_EXECUTOR_ATTRIBUTE_EXECUTOR_ID='$(SPARK_EXECUTOR_ID)'
spark.ui.custom.executor.log.url='https://log-server/log?appId={{APP_ID}}&execId={{
|
Looks like this works in master. Which versions before 4.0.0 support this? |
All Apache Spark with K8s GA have been supporting it. So, SPARK-49176 is a documentation fix. |
4 participants
What changes were proposed in this pull request?
Make Kubernetes resource manager support existing config
spark.ui.custom.executor.log.url.Allow for
Supports these variables:
APP_ID: The unique application idEXECUTOR_ID: The executor id (a positive integer larger than zero)HOSTNAME: The name of the host where the executor runsKUBERNETES_NAMESPACE: The namespace where the executor pods runKUBERNETES_POD_NAME: The name of the pod that contains the executorFILE_NAME: The name of the log, which is always"log"Why are the changes needed?
Running Spark on Kubernetes requires persisting the logs elsewhere. Having the Spark UI link to those logs is very useful. This is currently only supported by YARN.
Does this PR introduce any user-facing change?
Spark UI provides links to logs when run on Kubernetes.
How was this patch tested?
Unit test and manually tested on minikube K8S cluster.
Was this patch authored or co-authored using generative AI tooling?
No