Changed default in VPC Access connector throughput causes Terraform to re-create resource

[ahmetb]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.15.4
on darwin_amd64

• provider registry.terraform.io/hashicorp/google v3.71.0

Affected Resource(s)

• google_vpc_access_connector

Terraform Configuration Files

resource "google_vpc_access_connector" "default" {
  depends_on = [
    google_project_service.vpcaccess
  ]
  name          = "vpc-connector"
  region        = var.region
  project       = var.project
  network       = google_compute_network.vpc.name
  ip_cidr_range = "10.8.0.0/28"
}

// it's incomplete but we refer to this connector later as
// google_vpc_access_connector.default.name in cloud run service spec.

Debug Output

N/A

Panic Output

N/A

Expected Behavior

VPC Connector should not be deleted just because I upgraded my terraform google provider version.

Actual Behavior

Many of our customers use terraform apply -auto-approve in their headless CD environments.

So I just saw this during mine:

  # google_vpc_access_connector.default must be replaced
-/+ resource "google_vpc_access_connector" "default" {
      ~ id             = "projects/grpcoin/locations/us-west2/connectors/vpc-connector" -> (known after apply)
      ~ max_throughput = 1000 -> 300 # forces replacement
        name           = "vpc-connector"
      ~ self_link      = "projects/grpcoin/locations/us-west2/connectors/vpc-connector" -> (known after apply)
      ~ state          = "READY" -> (known after apply)
        # (5 unchanged attributes hidden)
    }

mirror (internal): http://screen/727AGd7ComADY3P

This means somewhere the default value is changed for throughput field.

Now the entire VPC connector has to be deleted and re-created, which takes many minutes, which means many minutes of downtime for customer apps because the lack of VPC connector means the apps will not work.

Furthermore, I am not sure what is the behavior for Cloud Run apps if the attached VPC Connector is disconnected and then created with the same name. Does it mean the app needs to get re-deployed? Because in Terraform, it will not be, since we refer to VPC connector "by name". (new Cloud Run instances are ok but this behavior is probably not defined for old

When using Terraform this should not happen. Ideally we should have default values for these disruptive resources that do not change.

Steps to Reproduce

1. terraform apply

output from command:

...

google_project_service.run: Refreshing state... [id=grpcoin/run.googleapis.com]
google_service_account.sa: Refreshing state... [id=projects/grpcoin/serviceAccounts/grpcoin@grpcoin.iam.gserviceaccount.com]
google_project_service.firestore: Refreshing state... [id=grpcoin/firestore.googleapis.com]
google_project_service.vpcaccess: Refreshing state... [id=grpcoin/vpcaccess.googleapis.com]
google_service_account.fe-sa: Refreshing state... [id=projects/grpcoin/serviceAccounts/grpcoin-fe@grpcoin.iam.gserviceaccount.com]
google_service_account.cron: Refreshing state... [id=projects/grpcoin/serviceAccounts/cron-pv@grpcoin.iam.gserviceaccount.com]
google_app_engine_application.app: Refreshing state... [id=grpcoin]
google_project_service.compute: Refreshing state... [id=grpcoin/compute.googleapis.com]
google_compute_url_map.https_redirect: Refreshing state... [id=projects/grpcoin/global/urlMaps/grpcoin-https-redirect]
google_project_service.scheduler: Refreshing state... [id=grpcoin/cloudscheduler.googleapis.com]
google_project_service.redis: Refreshing state... [id=grpcoin/redis.googleapis.com]
google_project.default: Refreshing state... [id=projects/grpcoin]
google_project_iam_binding.frontend-firestore-access: Refreshing state... [id=grpcoin/roles/datastore.viewer]
google_project_iam_binding.firestore-access: Refreshing state... [id=grpcoin/roles/datastore.user]
google_project_iam_binding.tracing-access: Refreshing state... [id=grpcoin/roles/cloudtrace.agent]
google_compute_target_http_proxy.https_redirect: Refreshing state... [id=projects/grpcoin/global/targetHttpProxies/lb-http-proxy]
google_compute_network.vpc: Refreshing state... [id=projects/grpcoin/global/networks/vpc]
google_compute_managed_ssl_certificate.default: Refreshing state... [id=projects/grpcoin/global/sslCertificates/grpcoin-cert]
google_compute_global_address.default: Refreshing state... [id=projects/grpcoin/global/addresses/lb-ip]
google_cloud_run_service.frontend: Refreshing state... [id=locations/us-west2/namespaces/grpcoin/services/grpcoin-frontend]
google_compute_global_forwarding_rule.https_redirect: Refreshing state... [id=projects/grpcoin/global/forwardingRules/lb-http-fwdrule]
google_vpc_access_connector.default: Refreshing state... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector]
google_redis_instance.cache: Refreshing state... [id=projects/grpcoin/locations/us-west2/instances/cache]
google_cloud_run_service_iam_policy.frontend-noauth: Refreshing state... [id=v1/projects/grpcoin/locations/us-west2/services/grpcoin-frontend]
google_cloud_scheduler_job.pv-job: Refreshing state... [id=projects/grpcoin/locations/us-west2/jobs/cron-portfolio-values]
google_compute_region_network_endpoint_group.frontend: Refreshing state... [id=projects/grpcoin/regions/us-west2/networkEndpointGroups/cr-fe-neg]
google_compute_backend_service.frontend: Refreshing state... [id=projects/grpcoin/global/backendServices/grpcoin-frontend]
google_cloud_run_service.apiserver: Refreshing state... [id=locations/us-west2/namespaces/grpcoin/services/grpcoin-main]
google_cloud_run_service_iam_policy.apiserver-noauth: Refreshing state... [id=v1/projects/grpcoin/locations/us-west2/services/grpcoin-main]
google_compute_region_network_endpoint_group.apiserver: Refreshing state... [id=projects/grpcoin/regions/us-west2/networkEndpointGroups/cr-api-neg]
google_compute_backend_service.apiserver: Refreshing state... [id=projects/grpcoin/global/backendServices/grpcoin-apiserver]
google_compute_url_map.default: Refreshing state... [id=projects/grpcoin/global/urlMaps/grpcoin-urlmap]
google_compute_target_https_proxy.default: Refreshing state... [id=projects/grpcoin/global/targetHttpsProxies/lb-https-proxy]
google_compute_global_forwarding_rule.default: Refreshing state... [id=projects/grpcoin/global/forwardingRules/lb-https-fwdrule]
Note: Objects have changed outside of Terraform
Terraform detected the following changes made outside of Terraform since the last "terraform apply":
google_compute_backend_service.apiserver has been changed
~ resource "google_compute_backend_service" "apiserver" {

+ custom_request_headers          = []

+ health_checks                   = []

id                              = "projects/grpcoin/global/backendServices/grpcoin-apiserver"

name                            = "grpcoin-apiserver"

# (12 unchanged attributes hidden)
  + backend {
      + balancing_mode               = "UTILIZATION"
      + capacity_scaler              = 1
      + group                        = "https://www.googleapis.com/compute/v1/projects/grpcoin/regions/us-west2/networkEndpointGroups/cr-api-neg"
      + max_connections              = 0
      + max_connections_per_endpoint = 0
      + max_connections_per_instance = 0
      + max_rate                     = 0
      + max_rate_per_endpoint        = 0
      + max_rate_per_instance        = 0
      + max_utilization              = 0
    }
  - backend {
      - balancing_mode  = "UTILIZATION" -> null
      - capacity_scaler = 1 -> null
      - group           = "projects/grpcoin/regions/us-west2/networkEndpointGroups/cr-api-neg" -> null
      - max_utilization = 0.8 -> null
    }
}

google_compute_managed_ssl_certificate.default has been changed
~ resource "google_compute_managed_ssl_certificate" "default" {

+ expire_time               = "2021-09-02T13:14:25.000-07:00"

id                        = "projects/grpcoin/global/sslCertificates/grpcoin-cert"

name                      = "grpcoin-cert"

~ subject_alternative_names = [

+ "grpco.in",

+ "api.grpco.in",

]

# (5 unchanged attributes hidden)
    # (1 unchanged block hidden)
}

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using

ignore_changes, the following plan may include actions to undo or respond to these changes.
────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated

with the following symbols:

-/+ destroy and then create replacement
Terraform will perform the following actions:
google_compute_global_address.default must be replaced
-/+ resource "google_compute_global_address" "default" {

~ address            = "34.102.163.199" -> (known after apply)

- address_type       = "EXTERNAL" -> null

~ creation_timestamp = "2021-06-04T13:15:40.297-07:00" -> (known after apply)

+ description        = "load balancer ip" # forces replacement

~ id                 = "projects/grpcoin/global/addresses/lb-ip" -> (known after apply)

name               = "lb-ip"

- prefix_length      = 0 -> null

~ project            = "grpcoin" -> (known after apply)

~ self_link          = "https://www.googleapis.com/compute/v1/projects/grpcoin/global/addresses/lb-ip" -> (known after apply)

}
google_project_iam_binding.frontend-firestore-access must be replaced
-/+ resource "google_project_iam_binding" "frontend-firestore-access" {

~ etag    = "BwXD4sRCZVU=" -> (known after apply)

~ id      = "grpcoin/roles/datastore.viewer" -> (known after apply)

~ role    = "roles/datastore.viewer" -> "roles/datastore.user" # forces replacement

# (2 unchanged attributes hidden)

}
google_vpc_access_connector.default must be replaced
-/+ resource "google_vpc_access_connector" "default" {

~ id             = "projects/grpcoin/locations/us-west2/connectors/vpc-connector" -> (known after apply)

~ max_throughput = 1000 -> 300 # forces replacement

name           = "vpc-connector"

~ self_link      = "projects/grpcoin/locations/us-west2/connectors/vpc-connector" -> (known after apply)

~ state          = "READY" -> (known after apply)

# (5 unchanged attributes hidden)

}
Plan: 3 to add, 0 to change, 3 to destroy.
Changes to Outputs:

~ lb_ip = "34.102.163.199" -> (known after apply)

google_project_iam_binding.frontend-firestore-access: Destroying... [id=grpcoin/roles/datastore.viewer]

google_vpc_access_connector.default: Destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector]

google_compute_global_address.default: Destroying... [id=projects/grpcoin/global/addresses/lb-ip]

google_project_iam_binding.frontend-firestore-access: Destruction complete after 8s

google_project_iam_binding.frontend-firestore-access: Creating...

google_compute_global_address.default: Still destroying... [id=projects/grpcoin/global/addresses/lb-ip, 10s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 10s elapsed]

google_compute_global_address.default: Destruction complete after 12s

google_compute_global_address.default: Creating...

google_project_iam_binding.frontend-firestore-access: Creation complete after 8s [id=grpcoin/roles/datastore.user]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 20s elapsed]

google_compute_global_address.default: Still creating... [10s elapsed]

google_compute_global_address.default: Creation complete after 11s [id=projects/grpcoin/global/addresses/lb-ip]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 30s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 40s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 50s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 1m0s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 1m10s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 1m20s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 1m30s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 1m40s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 1m50s elapsed]
google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 2m0s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 2m10s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 2m20s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 2m30s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 2m40s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 2m50s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 3m0s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 3m10s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 3m20s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 3m30s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 3m40s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 3m50s elapsed]

google_vpc_access_connector.default: Still destroying... [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector, 4m0s elapsed]

google_vpc_access_connector.default: Destruction complete after 4m1s

google_vpc_access_connector.default: Creating...

google_vpc_access_connector.default: Still creating... [10s elapsed]

google_vpc_access_connector.default: Still creating... [20s elapsed]

google_vpc_access_connector.default: Still creating... [30s elapsed]

google_vpc_access_connector.default: Still creating... [40s elapsed]

google_vpc_access_connector.default: Still creating... [50s elapsed]

google_vpc_access_connector.default: Still creating... [1m0s elapsed]

google_vpc_access_connector.default: Still creating... [1m10s elapsed]

google_vpc_access_connector.default: Still creating... [1m20s elapsed]

google_vpc_access_connector.default: Still creating... [1m30s elapsed]

google_vpc_access_connector.default: Still creating... [1m40s elapsed]

google_vpc_access_connector.default: Still creating... [1m50s elapsed]

google_vpc_access_connector.default: Still creating... [2m0s elapsed]

google_vpc_access_connector.default: Creation complete after 2m2s [id=projects/grpcoin/locations/us-west2/connectors/vpc-connector]

Important Factoids

N/A

References

Root cause seems to be this change: https://github.com/hashicorp/terraform-provider-google/blame/master/google/resource_vpc_access_connector.go#L63

It was done after an issue report #9228 however this fix causes a "new default" , which means the resource has to be deleted/re-created which results in downtime for user apps.

[edwardmedia]

@ahmetb sorry to hear about the inconvenience. I opened the PR for #9228. I have no change history of the field from api. And the provider does not have much control over the default values. But clearly it is now 300 (not 1000). Moving forward, we better use 300 or more users will face the issue of #9228. I understand your situation and that is a good case. Unfortunately I have not found a way to avoid this @c2thorn any thoughts?

[ahmetb]

https://github.com/hashicorp/terraform-provider-google/pull/9282/files and https://github.com/GoogleCloudPlatform/magic-modules/pull/4823/files seems off.

The rest api docs at https://cloud.google.com/vpc/docs/reference/vpcaccess/rest/v1/projects.locations.connectors#Connector and their internal proto versions both default to 200:

field	desc
minThroughput	integerMinimum throughput of the connector in Mbps. Default and min is 200.
maxThroughput	integerMaximum throughput of the connector in Mbps. Default is 200, max is 1000.

how did we come up with number 300?

Also why do these defaults need to be hardcoded? We keep changing these defaulting behavior in the API all the time (at least for Cloud Run api) and if TF is capable of merging server-defaulted values, we probably should not be hardcoding them in the provider?

[edwardmedia]

@ahmetb I noticed the api doc says it is 200. 300 is what the actual number from my api test

[ahmetb]

@edwardmedia that sounds like it should've been followed up by an internal bug first instead of making the change here. API reference docs which come from protos are supposed to be the source of truth, behavior can be buggy? But I agree I see the same behavior.

[edwardmedia]

@ahmetb I can't agree with you more about filing a bug against api. That is the root cause. In the provider code before the PR, it mentioned 200 was not correct.

[edwardmedia]

@ahmetb from the provider's perspective, I don't think there is much we can do for the time being. Closing the issue now

[ahmetb]

Confirming api docs were incorrect in this case, the default is 300.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.