Feature/add error handling for bqio by johnjcasey · Pull Request #30081 · apache/beam

johnjcasey · 2024-01-23T16:22:10Z

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

johnjcasey · 2024-01-23T17:24:28Z

Run Java PostCommit

…ning if errorhandler is used but error output isn't consumed.

github-actions · 2024-02-09T17:05:51Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

Abacn · 2024-02-14T15:47:31Z

"beam_PostCommit_Java" runs non-IO integration tests on direct runner. Would you mind trigger Dataflow PostCommit (which runs GCP IO integraiton tests on Dataflow) by adding file namedbeam_PostCommit_Java_DataflowV1.json ?

Abacn · 2024-02-14T17:16:51Z

I went through the changes and mostly looks straightforward and looks good to me. Due to the large number of line of changes I may missed something. Please run postcommit tests and see we do not break things (commented above). Also consider run Python xlang GCPIO Dataflow (trigger path beam_PostCommit_Python_Xlang_Gcp_Dataflow.json)

…qio' into feature/add-error-handling-for-bqio

ahmedabu98

Thanks, left some comments.
Nothing blocking but there are a few nits

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java

...orm/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamBundleSource.java

...-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java

...ud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiConvertMessages.java

...le-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java

Abacn · 2024-02-15T16:12:57Z

There are integration test failed:

org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOStorageReadIT > testBigQueryStorageRead1MErrorHandlingArrow FAILED
    java.lang.RuntimeException at BigQueryIOStorageReadIT.java:162

org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOStorageReadIT > testBigQueryStorageRead1MErrorHandlingAvro FAILED
    java.lang.RuntimeException at BigQueryIOStorageReadIT.java:162

See https://github.com/apache/beam/runs/21587463679 for details

johnjcasey · 2024-02-15T17:18:04Z

"beam_PostCommit_Java" runs non-IO integration tests on direct runner. Would you mind trigger Dataflow PostCommit (which runs GCP IO integraiton tests on Dataflow) by adding file namedbeam_PostCommit_Java_DataflowV1.json ?

Turns out the tests hadn't gotten run, and I missed something. Thanks for the catch!

…andling-for-bqio # Conflicts: # sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTranslation.java

ahmedabu98

Thanks, left a couple non-blocking comments

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java

...oud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageQueryIT.java

github-actions · 2024-02-20T23:34:53Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @damondouglas for label java.
R: @damccorm for label build.
R: @damondouglas for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

johnjcasey added 3 commits January 22, 2024 11:17

initial work on bqio dlq

78a7444

add test for exception handling on bq

7597a60

add tests

c09cacc

github-actions bot added java io gcp labels Jan 23, 2024

add trigger file to force the postcommit

c7ad419

github-actions bot added the build label Jan 23, 2024

wire error handling into storage write, update check to not log a war…

983b135

…ning if errorhandler is used but error output isn't consumed.

github-actions bot added runners dataflow labels Feb 1, 2024

johnjcasey added 4 commits February 1, 2024 13:07

wire batch loads with error handler

06f9096

wire in batch loads changes, update tests

df375df

add BQ write with error handling test cases

f001a04

spotless

921793b

johnjcasey marked this pull request as ready for review February 9, 2024 16:45

johnjcasey requested review from Abacn and ahmedabu98 February 9, 2024 16:45

Merge branch 'apache:master' into feature/add-error-handling-for-bqio

5130b38

johnjcasey added 2 commits February 14, 2024 13:44

add trigger files

e5c53fe

Merge remote-tracking branch 'origin/feature/add-error-handling-for-b…

27ad361

…qio' into feature/add-error-handling-for-bqio

ahmedabu98 reviewed Feb 15, 2024

View reviewed changes

Abacn mentioned this pull request Feb 15, 2024

Bump PostCommit Java DataflowV1 timeout to 6h #30323

Merged

3 tasks

Merge remote-tracking branch 'origin/master' into feature/add-error-h…

a2e4b50

…andling-for-bqio # Conflicts: # sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTranslation.java

johnjcasey added 2 commits February 15, 2024 12:32

address comments, support error handling when direct read from table

c6fb18b

revert change to build.gradle

c886de1

ahmedabu98 reviewed Feb 17, 2024

View reviewed changes

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java Outdated Show resolved Hide resolved

...oud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageQueryIT.java Show resolved Hide resolved

johnjcasey added 2 commits February 20, 2024 11:06

address comments, improve splitting logic

f78a73f

spotless

47fce17

github-actions bot added the Next Action: Reviewers label Feb 20, 2024

johnjcasey merged commit 6a8c27e into apache:master Feb 21, 2024

RustedBones mentioned this pull request Sep 4, 2024

BigQueryIO.read(SerializableFunction): Collect records that could not be parsed into the custom-typed object into a PCollection of TableRows #20704

Closed

Conversation

johnjcasey commented Jan 23, 2024

GitHub Actions Tests Status (on master branch)

Uh oh!

johnjcasey commented Jan 23, 2024

Uh oh!

github-actions bot commented Feb 9, 2024

Uh oh!

Abacn commented Feb 14, 2024

Uh oh!

Abacn commented Feb 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahmedabu98 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Abacn commented Feb 15, 2024

Uh oh!

johnjcasey commented Feb 15, 2024

Uh oh!

ahmedabu98 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Abacn commented Feb 14, 2024 •

edited

Loading