add tests and docs for using custom service account in dataflow flex template job

[xingao267]

fixes hashicorp/terraform-provider-google#7658

If this PR is for Terraform, I acknowledge that I have:

• Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
• Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
• Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
• Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
• Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

dataflow: added documentation about using `parameters` for custom service account and other pipeline options to `google_dataflow_flex_template_job`

[xingao267]

I tested it locally. It seems the dataflow jobs is created successfully with the custom service account, but test failed with the following error:

$ make testacc TEST=./google-beta TESTARGS='-run=TestAccDataflowFlexTemplateJob_withServiceAccount'
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta -v -run=TestAccDataflowFlexTemplateJob_withServiceAccount -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/version.ProviderVersion=acc"
=== RUN   TestAccDataflowFlexTemplateJob_withServiceAccount
=== PAUSE TestAccDataflowFlexTemplateJob_withServiceAccount
=== CONT  TestAccDataflowFlexTemplateJob_withServiceAccount
    provider_test.go:266: Step 1/1 error: Check failed: Check 2/2 error: Error getting dataflow job instance template: no instance template found for dataflow job "2020-11-24_09_30_40-2187942558452290776"
--- FAIL: TestAccDataflowFlexTemplateJob_withServiceAccount (783.03s)
FAIL
FAIL	github.com/hashicorp/terraform-provider-google-beta/google-beta	783.237s
FAIL
make: *** [GNUmakefile:15: testacc] Error 1

@rileykarson could you help take a look at this PR?

[rileykarson]

I think the error comes from testAccDataflowJobGetGeneratedInstanceTemplate. I'd suggest running the test with TF_LOG=DEBUG set, and looking at the HTTP requests/responses.

Also, can you add the field to the documentation? https://github.com/GoogleCloudPlatform/magic-modules/blob/master/third_party/terraform/website/docs/r/dataflow_flex_template_job.html.markdown

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 10 insertions(+), 4 deletions(-))
Terraform Beta: Diff ( 3 files changed, 205 insertions(+), 9 deletions(-))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=159774"

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 10 insertions(+), 4 deletions(-))
Terraform Beta: Diff ( 3 files changed, 209 insertions(+), 9 deletions(-))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=159774"

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 10 insertions(+), 4 deletions(-))
Terraform Beta: Diff ( 3 files changed, 206 insertions(+), 9 deletions(-))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=159779"

[modular-magician]

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDataSourceSpannerInstance_basic|TestAccActiveDirectoryDomainTrust_activeDirectoryDomainTrustBasicExample|TestAccBigqueryDataTransferConfig|TestAccComposerEnvironment_withUpdateOnCreate|TestAccComputeTargetInstance_targetInstanceCustomNetworkExample|TestAccFilestoreInstance_filestoreInstanceBasicExample|TestAccFilestoreInstance_update|TestAccProjectIamCustomRole_basic|TestAccOSLoginSSHPublicKey_osLoginSshKeyExpiry|TestAccSpannerDatabase_basic|TestAccStorageBucket_cors You can view the result here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=159828"

[xingao267]

I think the error comes from testAccDataflowJobGetGeneratedInstanceTemplate. I'd suggest running the test with TF_LOG=DEBUG set, and looking at the HTTP requests/responses.

Also, can you add the field to the documentation? https://github.com/GoogleCloudPlatform/magic-modules/blob/master/third_party/terraform/website/docs/r/dataflow_flex_template_job.html.markdown

Thanks! Fixed the test. For flex template job, I need to get the SA info directly from the VM but not VM template.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 10 insertions(+), 4 deletions(-))
Terraform Beta: Diff ( 3 files changed, 209 insertions(+), 9 deletions(-))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=159894"

[modular-magician]

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDataSourceSpannerInstance_basic|TestAccActiveDirectoryDomainTrust_activeDirectoryDomainTrustBasicExample|TestAccBigqueryDataTransferConfig|TestAccComposerEnvironment_withUpdateOnCreate|TestAccContainerNodePool_nodeLocations|TestAccContainerNodePool_basic|TestAccContainerNodePool_maxPodsPerNode|TestAccContainerNodePool_withInvalidUpgradeSettings|TestAccContainerNodePool_withNodeConfig|TestAccContainerNodePool_withGPU|TestAccContainerNodePool_withLinuxNodeConfig|TestAccContainerNodePool_withKubeletConfig|TestAccContainerNodePool_withNodeConfigScopeAlias|TestAccContainerNodePool_withSandboxConfig|TestAccContainerNodePool_withUpgradeSettings|TestAccContainerNodePool_withWorkloadIdentityConfig|TestAccContainerNodePool_regionalAutoscaling|TestAccContainerNodePool_withManagement|TestAccContainerNodePool_autoscaling|TestAccContainerNodePool_regionalClusters|TestAccContainerNodePool_shieldedInstanceConfig|TestAccContainerNodePool_resize|TestAccContainerNodePool_012_ConfigModeAttr|TestAccContainerNodePool_EmptyGuestAccelerator|TestAccFilestoreInstance_filestoreInstanceBasicExample|TestAccFilestoreInstance_update|TestAccOSLoginSSHPublicKey_osLoginSshKeyExpiry You can view the result here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=159896"

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 8 insertions(+), 5 deletions(-))
Terraform Beta: Diff ( 2 files changed, 180 insertions(+), 10 deletions(-))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=160866"

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 8 insertions(+), 5 deletions(-))
Terraform Beta: Diff ( 2 files changed, 180 insertions(+), 10 deletions(-))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=160868"

[rileykarson]

Looking at the "all commits" view, it appears that the actual resource changes got reverted at some point?

[xingao267]

@rileykarson according to hashicorp/terraform-provider-google#7658 (comment), it turned out we can specify custom SA in the parameters block. So I reverted the actual resource changes and made this PR just adding tests and docs for how to use custom SA.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 8 insertions(+), 5 deletions(-))
Terraform Beta: Diff ( 2 files changed, 182 insertions(+), 10 deletions(-))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=160870"

[modular-magician]

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDataSourceComposerEnvironment_basic|TestAccDataSourceSpannerInstance_basic|TestAccActiveDirectoryDomainTrust_activeDirectoryDomainTrustBasicExample|TestAccBigqueryDataTransferConfig|TestAccComposerEnvironment_withUpdateOnCreate|TestAccFilestoreInstance_filestoreInstanceBasicExample|TestAccFilestoreInstance_update|TestAccOSLoginSSHPublicKey_osLoginSshKeyExpiry You can view the result here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=160872"

[modular-magician]

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDataSourceComposerEnvironment_basic|TestAccDataSourceSpannerInstance_basic|TestAccActiveDirectoryDomainTrust_activeDirectoryDomainTrustBasicExample|TestAccBigqueryDataTransferConfig|TestAccComposerEnvironment_withUpdateOnCreate|TestAccFilestoreInstance_filestoreInstanceBasicExample|TestAccFilestoreInstance_update|TestAccOSLoginSSHPublicKey_osLoginSshKeyExpiry You can view the result here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=160872"

[rileykarson]

LGTM- just running through the test pipeline before merging.

/gcbrun

[rileykarson]

Well, that didn't appear to have taken. I'll try again!

/gcbrun

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 8 insertions(+), 5 deletions(-))
Terraform Beta: Diff ( 2 files changed, 182 insertions(+), 10 deletions(-))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=161491"

[xingao267]

@rileykarson I can't see the builds, do you know why it is failing?

[rileykarson]

Just a timeout! The VCR test run I triggered should supersede it. It doesn't look like the second phase kicked off- I'll do that manually on Monday.

[xingao267]

@rileykarson any updates on this?

[rileykarson]

Sorry! Heavy load of interrupts on my part this last week kept me distracted. Started a run now.

For my own benefit, that's https://ci-oss.hashicorp.engineering/buildConfiguration/GoogleCloudBeta_ProviderGoogleCloudBetaMmUpstream/162586?buildTab=overview

[xingao267]

@rileykarson it looks like the documentation change included in this PR did not get released to https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dataflow_flex_template_job. Did I miss anything?

[rileykarson]

That's expected! We had a fairly substantial gap between releases (hashicorp/terraform-provider-google#7998)

It should be available today.

[xingao267]

@rileykarson got it, thanks for explaining!