Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.
Description
When scaling down a dataproc cluster w/ terraform by updating the number of worker nodes, currently the instances are immediately terminated. If there is work being done on these nodes that work is lost and may lead to job failures. Dataproc API has an option for graceful decomissioning. There is a gracefulDecomissionTimeout parameter that should be exposed as part of the resource and when updating the number of nodes (this is only really important when scaling down).
Right now to safely scale down an cluster w/ some jobs running you have to manually make an update request (thus diverging from your TF state which is obviously problematic for the next TF apply).
As we add support terraform for Dataproc AutoScaling Policies, where one can also set a graceful decomissioning timeout this creates a potential for a confusing interface. For non-autoscaling clusters there is a strong need for this as a top level property. However, it should be documented that when each would be respected. For example the top level graceful decomissioning timeout should apply to actions taken by terraform due to an update to the number of workers. The timeout listed w/ in the autoscaling policy would dictate the timeout to be used when the dataproc service autoscales your cluster.
New or Affected Resource(s)
Potential Terraform Configuration
# Propose what you think the configuration to take advantage of this feature should look like.
# We may not use it verbatim, but it's helpful in understanding your intent.
resource "google_dataproc_cluster" "simplecluster" {
...
# This should be used whenever issuing requests to scale down the cluster defined in this resource.
graceful_decomission_timeout = "90m"
...
}References
Community Note
Description
When scaling down a dataproc cluster w/ terraform by updating the number of worker nodes, currently the instances are immediately terminated. If there is work being done on these nodes that work is lost and may lead to job failures. Dataproc API has an option for graceful decomissioning. There is a
gracefulDecomissionTimeoutparameter that should be exposed as part of the resource and when updating the number of nodes (this is only really important when scaling down).Right now to safely scale down an cluster w/ some jobs running you have to manually make an update request (thus diverging from your TF state which is obviously problematic for the next TF apply).
As we add support terraform for Dataproc AutoScaling Policies, where one can also set a graceful decomissioning timeout this creates a potential for a confusing interface. For non-autoscaling clusters there is a strong need for this as a top level property. However, it should be documented that when each would be respected. For example the top level graceful decomissioning timeout should apply to actions taken by terraform due to an update to the number of workers. The timeout listed w/ in the autoscaling policy would dictate the timeout to be used when the dataproc service autoscales your cluster.
New or Affected Resource(s)
Potential Terraform Configuration
References