add iam workload identity federation

[wvanderdeijl]

This is my attempt to add the new Google Workload Identity Federation resources.

fixes hashicorp/terraform-provider-google#7455

But I run into the issue that the IAM resources seem to be handwritten. Now that I add products/iam/terraform.yaml all the resources in that file get generated to terraform, including things like service_account and role.
What would be the best way to leave the handwritten iam resources alone, but still be able to add the new iam-beta resources for workload identity federation?

If this PR is for Terraform, I acknowledge that I have:

• Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
• Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
• Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
• Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
• Read the Release Notes Guide before writing my release note below.

`google_iam_workload_identity_pool`

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@rileykarson, please review this PR or find an appropriate assignee.

[rileykarson]

The {{id}} field will also need to appear as a field under properties (or parameters, but being in that list instead doesn't actually do anything).

You can then add url_param_only: true to indicate that it's only used in the URL. See

magic-modules/products/servicedirectory/api.yaml

Lines 50 to 57 in 6c2be6a

	- !ruby/object:Api::Type::String
	name: 'location'
	description: |
	The location for the Namespace.
	A full list of valid locations can be found by running
	`gcloud beta service-directory locations list`.
	required: true
	url_param_only: true

for an example.

I'd also recommend avoiding the id name, just because Terraform treats id as a reserved field name. workloadIdentityPoolId from https://cloud.google.com/iam/docs/reference/rest/v1beta/projects.locations.workloadIdentityPools/create seems appropriate.

[rileykarson]

Interesting- Magic Modules assumes that subsequent APIs are a rough superset of each other, but this v1beta API doesn't contain the other handful of IAM resources at all. We may need to represent this as a separate entry in the products/ folder, and override it in terraform.yaml to name it correctly.

You don't need that yet- I want to see what changes the generator makes, exactly.

[rileykarson]

Ah, yep, we'll want to add this as a new products/ entry. MM assumes that each version in an API is a rough superset of more stable versions (ga < beta < alpha < eap) because it uses a single API endpoint for all resources under a product. IAM exposes some beta-only resources at a beta endpoint but doesn't also have the GA resources there.

You'll want to create a new products/ folder- probably iambeta- and create an api.yaml and terraform.yaml there. The name (line 21 in this file) should also likely be IAMBeta. Inside terraform.yaml, you can override the default resource name using https://github.com/GoogleCloudPlatform/magic-modules/blob/master/overrides/terraform/resource_override.rb#L34-L38 so that the resources are google_iam_* not google_iam_beta_*

This should result in a second IAM basepath/endpoint getting registered, which points to the beta endpoint.

[rileykarson]

Would you mind breaking this resource into a followup PR afterwards? In my experience it tends to be a lot easier if we're iterating on one resource at a time, especially because a lot of comments that apply to one resource will apply to the second.

[wvanderdeijl]

I've removed WorkloadIdentityPoolProvider for now and we can add that in a subsequent PR

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 68 insertions(+), 16 deletions(-))
Terraform Beta: Diff ( 9 files changed, 739 insertions(+), 18 deletions(-))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 13 deletions(-))
Inspec: Diff ( 16 files changed, 80 insertions(+), 16 deletions(-))

[wvanderdeijl]

I added a first attempt at a test, but can't figure out why this is failing when running the terraform-provider-google-beta tests:

terraform-provider-google-beta % make testacc TEST=./google-beta TESTARGS='-run=TestAccIAMWorkloadIdentityPool'
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta -v -run=TestAccIAMWorkloadIdentityPool -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/version.ProviderVersion=acc"
=== RUN   TestAccIAMWorkloadIdentityPool_example
=== PAUSE TestAccIAMWorkloadIdentityPool_example
=== CONT  TestAccIAMWorkloadIdentityPool_example
2020/10/09 15:40:12 [WARN] Got error running Terraform: 2020/10/09 15:40:11 [DEBUG] Using modified User-Agent: Terraform/0.12.24 HashiCorp-terraform-exec/0.10.0

Error: Invalid resource type

  on ../../../../../../../../var/folders/wf/xm7lxbtn1nbbvzfzb3r4v89r0000gn/T/tftest228702188/work766105179/config285216703/terraform_plugin_test.tf line 13, in resource "google_iam_workload_identity_pool" "my_pool":
  13: resource "google_iam_workload_identity_pool" "my_pool" {

The provider provider.google does not support resource type
"google_iam_workload_identity_pool".

    TestAccIAMWorkloadIdentityPool_example: provider_test.go:266: Step 1/2 error: Error running pre-apply refresh: 2020/10/09 15:40:11 [DEBUG] Using modified User-Agent: Terraform/0.12.24 HashiCorp-terraform-exec/0.10.0
        
        Error: Invalid resource type
        
          on ../../../../../../../../var/folders/wf/xm7lxbtn1nbbvzfzb3r4v89r0000gn/T/tftest228702188/work766105179/config285216703/terraform_plugin_test.tf line 13, in resource "google_iam_workload_identity_pool" "my_pool":
          13: resource "google_iam_workload_identity_pool" "my_pool" {
        
        The provider provider.google does not support resource type
        "google_iam_workload_identity_pool".
        
--- FAIL: TestAccIAMWorkloadIdentityPool_example (4.01s)
FAIL
FAIL    github.com/hashicorp/terraform-provider-google-beta/google-beta 5.022s
FAIL
make: *** [testacc] Error 1

This is weird as I can see that google_iam_workload_identity_pool is generated in the list of resources in provider.go

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 75 insertions(+), 23 deletions(-))
Terraform Beta: Diff ( 9 files changed, 746 insertions(+), 25 deletions(-))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 13 deletions(-))
Inspec: Diff ( 16 files changed, 80 insertions(+), 16 deletions(-))

[rileykarson]

Sorry for the delay in getting back!

The error you're getting is because the tests are being run in the GA provider and not the google-beta provider. The GA google provider doesn't support beta features.

You'll want to supply -v beta to MM to generate into the terraform-google-provider-beta repo.

[rileykarson]

Ah, yep, we'll want to add this as a new products/ entry. MM assumes that each version in an API is a rough superset of more stable versions (ga < beta < alpha < eap) because it uses a single API endpoint for all resources under a product. IAM exposes some beta-only resources at a beta endpoint but doesn't also have the GA resources there.

You'll want to create a new products/ folder- probably iambeta- and create an api.yaml and terraform.yaml there. The name (line 21 in this file) should also likely be IAMBeta. Inside terraform.yaml, you can override the default resource name using https://github.com/GoogleCloudPlatform/magic-modules/blob/master/overrides/terraform/resource_override.rb#L34-L38 so that the resources are google_iam_* not google_iam_beta_*

This should result in a second IAM basepath/endpoint getting registered, which points to the beta endpoint.

[wvanderdeijl]

I've moved the new resources to products/iambeta/api.yaml and running bundle exec compiler -v "beta" -e terraform -o "$GOPATH/src/github.com/terraform-providers/terraform-provider-google-beta" -a seems to generate okay source code for terraform. This also includes my (beta) tests, but unfortunately I still cannot run the test (not sure why):

make testacc TEST=./google-beta TESTARGS='-run=TestAccIAMBetaWorkloadIdentityPool_example'
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta -v -run=TestAccIAMBetaWorkloadIdentityPool_example -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/version.ProviderVersion=acc"
=== RUN   TestAccIAMBetaWorkloadIdentityPool_example
=== PAUSE TestAccIAMBetaWorkloadIdentityPool_example
=== CONT  TestAccIAMBetaWorkloadIdentityPool_example
    TestAccIAMBetaWorkloadIdentityPool_example: provider_test.go:266: Step 1/1 error: Error running pre-apply refresh: 2020/10/13 21:52:00 [DEBUG] Using modified User-Agent: Terraform/0.12.24 HashiCorp-terraform-exec/0.10.0
        
        Error: Invalid resource type
        
          on ../../../../../../../../var/folders/wf/xm7lxbtn1nbbvzfzb3r4v89r0000gn/T/tftest637359650/work048036121/config035092829/terraform_plugin_test.tf line 13, in resource "google_iam_workload_identity_pool" "my_pool":
          13: resource "google_iam_workload_identity_pool" "my_pool" {
        
        The provider provider.google does not support resource type
        "google_iam_workload_identity_pool".

But I can see the new google_iam_workload_identity_pool resource is being generated into google-beta/provider.go so not really sure what is going there

Add the google-beta provider to the example also doesn't work:

resource "google_iam_workload_identity_pool" "my_pool" {
  provider     = google-beta

gives

=== CONT  TestAccIAMBetaWorkloadIdentityPool_example
    TestAccIAMBetaWorkloadIdentityPool_example: provider_test.go:266: Step 1/1 error: Error running pre-apply refresh: 2020/10/13 21:57:50 [DEBUG] Using modified User-Agent: Terraform/0.12.24 HashiCorp-terraform-exec/0.10.0
        
        Error: Could not satisfy plugin requirements
        
        
        Plugin reinitialization required. Please run "terraform init".
        
        Plugins are external binaries that Terraform uses to access and manipulate
        resources. The configuration provided requires plugins which can't be located,
        don't satisfy the version constraints, or are otherwise incompatible.
        
        Terraform automatically discovers provider requirements from your
        configuration, including providers used in child modules. To see the
        requirements and constraints from each module, run "terraform providers".
        
        
        
        Error: provider.google-beta: no suitable version installed
          version requirements: "(any version)"
          versions installed: none

Is there another beta-only resource with a test I can copy from?

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 4 files changed, 127 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 8 files changed, 698 insertions(+), 3 deletions(-))

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 125 insertions(+))
Terraform Beta: Diff ( 7 files changed, 696 insertions(+), 2 deletions(-))

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 125 insertions(+))
Terraform Beta: Diff ( 7 files changed, 696 insertions(+), 2 deletions(-))

[wvanderdeijl]

Most changes to identity-pools (and providers) return Operations that we need to poll for. I can copy that from other similar resources, but would like to get the test running before attempting so. With a (failing) test it's much easier for me to iterate on attempts to get it all working.

[rileykarson]

Are you still encountering the same issue with the branch as it is now? I received the following, which indicates the rest ran successfully:

------- Stdout: -------
=== RUN   TestAccIAMBetaWorkloadIdentityPool_example
=== PAUSE TestAccIAMBetaWorkloadIdentityPool_example
=== CONT  TestAccIAMBetaWorkloadIdentityPool_example
TestAccIAMBetaWorkloadIdentityPool_example: provider_test.go:266: Step 1/1 error: Error running pre-apply refresh: 2020/10/14 21:29:32 [DEBUG] Using modified User-Agent: Terraform/0.12.29 HashiCorp-terraform-exec/0.10.0
Error: Missing required argument
on config516010655/terraform_plugin_test.tf line 13, in resource "google_iam_workload_identity_pool" "my_pool":
13: resource "google_iam_workload_identity_pool" "my_pool" {
The argument "workload_identity_pool_id" is required, but no definition was
found.
--- FAIL: TestAccIAMBetaWorkloadIdentityPool_example (0.39s)
FAIL

One thing you could try is checking out https://github.com/modular-magician/terraform-provider-google-beta/tree/auto-pr-4071, the code generated by our CI by your current changes (and the code I ran), and then running MM on top of that locally.

[wvanderdeijl]

I must be doing something wrong with starting the tests. I've checkout your branch https://github.com/modular-magician/terraform-provider-google-beta/tree/auto-pr-4071 and tried to run the tests and run into the same errors:

% make testacc TEST=./google-beta TESTARGS='-run=TestAccIAMBetaWorkloadIdentityPool_example'
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta -v -run=TestAccIAMBetaWorkloadIdentityPool_example -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/version.ProviderVersion=acc"
=== RUN   TestAccIAMBetaWorkloadIdentityPool_example
=== PAUSE TestAccIAMBetaWorkloadIdentityPool_example
=== CONT  TestAccIAMBetaWorkloadIdentityPool_example
    TestAccIAMBetaWorkloadIdentityPool_example: provider_test.go:266: Step 1/1 error: Error running pre-apply refresh: 2020/10/15 15:07:25 [DEBUG] Using modified User-Agent: Terraform/0.12.24 HashiCorp-terraform-exec/0.10.0
        
        Error: Invalid resource type
        
          on ../../../../../../../../var/folders/wf/xm7lxbtn1nbbvzfzb3r4v89r0000gn/T/tftest422138618/work352437777/config639766485/terraform_plugin_test.tf line 13, in resource "google_iam_workload_identity_pool" "my_pool":
          13: resource "google_iam_workload_identity_pool" "my_pool" {
        
        The provider provider.google does not support resource type
        "google_iam_workload_identity_pool".
        
--- FAIL: TestAccIAMBetaWorkloadIdentityPool_example (3.13s)
FAIL
FAIL    github.com/hashicorp/terraform-provider-google-beta/google-beta 3.803s
FAIL
make: *** [testacc] Error 1

This is using the test command I got from the contribution guide at https://github.com/hashicorp/terraform-provider-google/blob/master/.github/CONTRIBUTING.md:

make testacc TEST=./google TESTARGS='-run=TestAccContainerNodePool_basic'

The only thing I changed is the TEST argument to ./google-beta since I am testing the beta provider. Is there something I'm missing?

Nevermind, I forgot to upgrade terraform to the latest version. Now upgrade to v0.13.4 and the test runs (and fails as expected). While I continu to work on this PR I'll mark it as draft until it is complete

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 125 insertions(+))
Terraform Beta: Diff ( 7 files changed, 696 insertions(+), 2 deletions(-))

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 4 files changed, 165 insertions(+))
Terraform Beta: Diff ( 10 files changed, 1047 insertions(+), 2 deletions(-))
TF OiCS: Diff ( 8 files changed, 213 insertions(+))

[wvanderdeijl]

I have added some tests and made some changes to the code to ensure all tests succeed. I think this is ready for review by @rileykarson

[wvanderdeijl]

Should I also make sure a terraform data source is generated? (if so, what are the basic steps to get a data source - are they all handcoded or somehow generated?)

[rileykarson]

LGTM!

If you're interested in adding a datasource, #4111 is a good template for what adding one looks like. They're handwritten, but based off the existing resource code. I'd suggest doing that in a separate PR (including from the one adding WorkloadIdentityPoolProvider) for ease of review.