From 9963ce6aefb14b43dc180c8d65d1787d6936db2d Mon Sep 17 00:00:00 2001 From: Maciej Obuchowski Date: Wed, 8 Jul 2026 18:26:30 +0200 Subject: [PATCH] Document GlueJobOperator support for OpenLineage Spark injection The GlueJobOperator has supported automatic OpenLineage parent job and transport information injection since it was added to the operator, but the Spark integration guide never listed it, leaving users unaware that the same [openlineage] settings apply to Glue jobs. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Maciej Obuchowski --- providers/openlineage/docs/spark.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/providers/openlineage/docs/spark.rst b/providers/openlineage/docs/spark.rst index 0dbdf967c142d..6e8647bddc969 100644 --- a/providers/openlineage/docs/spark.rst +++ b/providers/openlineage/docs/spark.rst @@ -94,6 +94,7 @@ to manually configure these properties in every Spark operator. Automatic injection is supported for the following operators: +- :class:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator` - :class:`~airflow.providers.apache.livy.operators.livy.LivyOperator` - :class:`~airflow.providers.apache.spark.operators.spark_submit.SparkSubmitOperator` - :class:`~airflow.providers.databricks.operators.databricks.DatabricksSubmitRunOperator` @@ -101,6 +102,14 @@ Automatic injection is supported for the following operators: - :class:`~airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateInlineWorkflowTemplateOperator` - :class:`~airflow.providers.google.cloud.operators.dataproc.DataprocSubmitJobOperator` +.. note:: + + :class:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator` receives the injected properties + through the Glue job's ``--conf`` argument (in ``script_args``) rather than a Spark ``conf`` dictionary. + As with the other operators, injection only propagates the parent job and transport information; the Glue + job itself must still have the Spark OpenLineage integration (the ``OpenLineageSparkListener``) enabled for + lineage to be emitted. + .. _options:spark_inject_parent_job_info: