Skip to content

Log SQL query on its own line in BigQueryInsertJobOperator#69386

Merged
shahar1 merged 1 commit into
apache:mainfrom
AlejandroMorgante:improve-bigquery-insert-job-logging
Jul 6, 2026
Merged

Log SQL query on its own line in BigQueryInsertJobOperator#69386
shahar1 merged 1 commit into
apache:mainfrom
AlejandroMorgante:improve-bigquery-insert-job-logging

Conversation

@AlejandroMorgante

@AlejandroMorgante AlejandroMorgante commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

When BigQueryInsertJobOperator starts, it logs the whole job configuration dict in a single line. For query jobs this makes the SQL essentially unreadable in the task log — the query shows up embedded in the dict repr with escaped \n sequences:

Executing: {'query': {'query': 'select *\nfrom table\nwhere condition', 'useLegacySql': False}}'

This change logs the SQL on its own log line (reusing the operator's existing sql property) so newlines render naturally and the query can be read — and copy-pasted into the BigQuery console — directly from the log:

Executing: {'query': {'query': 'select *\nfrom table\nwhere condition', 'useLegacySql': False}}
SQL query:
select *
from table
where condition

This restores a behaviour Airflow used to have rather than adding something new: BigQueryExecuteQueryOperator logged the plain SQL string directly — self.log.info("Executing: %s", self.sql) — until the deprecated operator was removed in #43953, leaving the configuration-dict dump as the only trace of the query in the logs. This also matches how sibling operators log SQL today (e.g. BigQueryCheckOperator's Executing SQL check: %s). The stray trailing quote in the current log line is fixed along the way.

Readable SQL in the log matters most after a failure or when logs are consumed remotely (CloudWatch/Stackdriver/grep), where the UI's rendered-template view isn't available. For non-query job types (load/copy/extract) the behaviour is unchanged.

No test added: the change is log-output-only, and the project testing standards discourage asserting on log output.

Verification performed:

  • All 37 BigQueryInsertJobOperator unit tests pass in Breeze; mypy and static checks clean.
  • Ran the example_bigquery_value_check.py system test end-to-end against a real GCP project: the Dag passed (1 passed in 188s), and the task log shows the new SQL query: line rendered as plain text.

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Fable 5)

Generated-by: Claude Code (Fable 5) following the guidelines

Co-authored-by: Lucho Formisano <22617963+LuchoFormisano@users.noreply.github.com>
@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Jul 4, 2026
@AlejandroMorgante

Copy link
Copy Markdown
Contributor Author

Verified the change end-to-end against a real GCP project. Same DAG with a multiline query, run twice: first with the provider built from main (before) and then from this PR's head (eb53faf, after). Before: the SQL is only visible inside the single-line config dict with escaped \n. After: the new SQL query: line renders the query on real lines, readable and copy-pasteable straight from the task log. Job inserted and run succeeded in both cases — screenshots attached.

01_before 02_after 03_run_overview

@AlejandroMorgante AlejandroMorgante marked this pull request as ready for review July 4, 2026 17:04
@shahar1

shahar1 commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

CC: @VladaZakharova @MaksYermak , WDYT?

@shahar1 shahar1 merged commit ffe4fcf into apache:main Jul 6, 2026
160 of 162 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants