Skip to content

Add BigTable source format in BigQuery tables #4155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
slevenick merged 15 commits into from
Apr 15, 2021
Merged

Add BigTable source format in BigQuery tables #4155

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
bf4a252
Add BigTable source format in BigQuery
jmthvt Oct 28, 2020
828eb0a
Add tests
jmthvt Oct 28, 2020
f503762
Fix test config typos
jmthvt Oct 29, 2020
4e6bf18
Merge branch 'master' into feat/add-bigtable-sourceformat
jmthvt Oct 30, 2020
0fdd96b
Add bigtable external source in the bigquery tests
jmthvt Nov 2, 2020
137dff6
Merge branch 'feat/add-bigtable-sourceformat' of github.com:jmthvt/ma…
jmthvt Nov 2, 2020
d9a0a51
typo
jmthvt Nov 13, 2020
a1c8a93
Merge branch 'master' into feat/add-bigtable-sourceformat
jmthvt Nov 24, 2020
05336e5
Shrink resource name
jmthvt Nov 24, 2020
552aa0e
Fix master merge
jmthvt Nov 25, 2020
e992d4e
Merge master & resolve test conflict
jmthvt Mar 30, 2021
7c848b2
Merge branch 'master' into feat/add-bigtable-sourceformat
jmthvt Apr 12, 2021
7eb6f1d
Merge branch 'master' into feat/add-bigtable-sourceformat
jmthvt Apr 13, 2021
194aeb7
Merge branch 'master' into feat/add-bigtable-sourceformat
jmthvt Apr 15, 2021
2973a68
Ignore deletion_protection in TestAccBigQueryDataTable_bigtable
jmthvt Apr 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion products/bigquery/api.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -722,7 +722,8 @@ objects:
description: |
The format of the data files. For CSV files, specify "CSV". For datastore backups, specify "DATASTORE_BACKUP".
For newline-delimited JSON, specify "NEWLINE_DELIMITED_JSON". For Avro, specify "AVRO". For parquet, specify "PARQUET".
For orc, specify "ORC". The default value is CSV.
For orc, specify "ORC". [Beta] For Bigtable, specify "BIGTABLE".
The default value is CSV.
default_value: 'CSV'
- !ruby/object:Api::Type::Boolean
name: 'allowJaggedRows'
Expand Down
4 changes: 2 additions & 2 deletions third_party/terraform/resources/resource_bigquery_table.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,9 +153,9 @@ func resourceBigQueryTable() *schema.Resource {
"source_format": {
Type: schema.TypeString,
Required: true,
Description: `The data format. Supported values are: "CSV", "GOOGLE_SHEETS", "NEWLINE_DELIMITED_JSON", "AVRO", "PARQUET", and "DATSTORE_BACKUP". To use "GOOGLE_SHEETS" the scopes must include "googleapis.com/auth/drive.readonly".`,
Description: `The data format. Supported values are: "CSV", "GOOGLE_SHEETS", "NEWLINE_DELIMITED_JSON", "AVRO", "PARQUET", "DATSTORE_BACKUP", and "BIGTABLE". To use "GOOGLE_SHEETS" the scopes must include "googleapis.com/auth/drive.readonly".`,
Copy link
Copy Markdown
Contributor

@venkykuberan venkykuberan Nov 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder ORC can also included in the source_format list ?. We have an open issue (hashicorp/terraform-provider-google#7691) for that type.

Copy link
Copy Markdown
Contributor Author

@jmthvt jmthvt Nov 10, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, sorry but that's out of scope for this PR - adding it in the list is trivial but requires to write the tests.

ValidateFunc: validation.StringInSlice([]string{
"CSV", "GOOGLE_SHEETS", "NEWLINE_DELIMITED_JSON", "AVRO", "DATSTORE_BACKUP", "PARQUET",
"CSV", "GOOGLE_SHEETS", "NEWLINE_DELIMITED_JSON", "AVRO", "DATSTORE_BACKUP", "PARQUET", "BIGTABLE",
}, false),
},
// SourceURIs [Required] The fully-qualified URIs that point to your data in Google Cloud.
Expand Down
55 changes: 55 additions & 0 deletions third_party/terraform/tests/resource_bigquery_table_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,6 +416,30 @@ func TestAccBigQueryDataTable_sheet(t *testing.T) {
})
}

func TestAccBigQueryDataTable_bigtable(t *testing.T) {
Copy link
Copy Markdown
Contributor

@slevenick slevenick Oct 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test should only be run for the beta provider, right? This file may need to be renamed to end in .go.erb and wrap this test in if/else tags to check the version.

An example can be seen here: https://github.com/GoogleCloudPlatform/magic-modules/blob/master/third_party/terraform/tests/resource_binaryauthorization_policy_test.go.erb#L41

Copy link
Copy Markdown
Contributor Author

@jmthvt jmthvt Oct 30, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's quite a weird one actually:
As per my understanding, you can set the format to BIGTABLE in stable (search for sourceFormat in https://bigquery.googleapis.com/discovery/v1/apis/bigquery/v2/rest )
However only 2 regions are supported for the queries https://cloud.google.com/bigquery/external-data-bigtable#supported_regions_and_zones.

From the BigQuery API point of view, I don't think it requires the beta provider.
But from BigTable it does.
Not sure of the preferred approach here.

t.Parallel()

context := map[string]interface{}{
"random_suffix": randString(t, 10),
}

vcrTest(t, resource.TestCase{
PreCheck: func() { testAccPreCheck(t) },
Providers: testAccProviders,
CheckDestroy: testAccCheckBigQueryTableDestroyProducer(t),
Steps: []resource.TestStep{
{
Config: testAccBigQueryTableFromBigtable(context),
Copy link
Copy Markdown
Contributor

@slevenick slevenick Oct 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this doesn't match the function declaration, causing this error:
google-beta/resource_bigquery_table_test.go:432:13: undefined: testAccBigQueryTableFromBigtable

Copy link
Copy Markdown
Contributor Author

@jmthvt jmthvt Oct 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, thanks!
Should be fixed now.

},
{
ResourceName: "google_bigquery_table.table",
ImportState: true,
ImportStateVerify: true,
},
},
})
}

func testAccCheckBigQueryExtData(t *testing.T, expectedQuoteChar string) resource.TestCheckFunc {
return func(s *terraform.State) error {
for _, rs := range s.RootModule().Resources {
Expand Down Expand Up @@ -1048,6 +1072,37 @@ func testAccBigQueryTableFromSheet(context map[string]interface{}) string {
`, context)
}

func testAccBigQueryTableFromBigtable(context map[string]interface{}) string {
return Nprintf(`
resource "google_bigquery_table" "table" {
dataset_id = google_bigquery_dataset.dataset.dataset_id
table_id = "tf_test_bigtable_%{random_suffix}"

external_data_configuration {
autodetect = true
source_format = "BIGTABLE"
ignore_unknown_values = true

source_uris = [
"https://googleapis.com/bigtable/projects/project_id/instances/instance_id/tables/table_name",
Copy link
Copy Markdown
Contributor

@slevenick slevenick Oct 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is causing an error during testing. This is an invalid source, maybe try using one of the source_uris from another test?

Copy link
Copy Markdown
Contributor Author

@jmthvt jmthvt Oct 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try to create a bigtable in the test & generate the URI.

Copy link
Copy Markdown
Contributor Author

@jmthvt jmthvt Nov 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@slevenick that's added but I'm unable to run the tests - can you please try again?

]
}
}

resource "google_bigquery_dataset" "dataset" {
dataset_id = "tf_test_ds_%{random_suffix}"
friendly_name = "test"
description = "This is a test description"
location = "EU"
default_table_expiration_ms = 3600000

labels = {
env = "default"
}
}
`, context)
}

var TEST_CSV = `lifelock,LifeLock,,web,Tempe,AZ,1-May-07,6850000,USD,b
lifelock,LifeLock,,web,Tempe,AZ,1-Oct-06,6000000,USD,a
lifelock,LifeLock,,web,Tempe,AZ,1-Jan-08,25000000,USD,c
Expand Down
2 changes: 1 addition & 1 deletion third_party/terraform/website/docs/r/bigquery_table.html.markdown
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ The `external_data_configuration` block supports:

* `source_format` (Required) - The data format. Supported values are:
"CSV", "GOOGLE_SHEETS", "NEWLINE_DELIMITED_JSON", "AVRO", "PARQUET",
and "DATSTORE_BACKUP". To use "GOOGLE_SHEETS"
"DATSTORE_BACKUP", and "BIGTABLE". To use "GOOGLE_SHEETS"
the `scopes` must include
"https://www.googleapis.com/auth/drive.readonly".

Expand Down