Tracking Remote State Lineage to Reduce Mistakes

[apparentlymart]

In the situation where a state is already present locally (either in terraform.tfstate or .terraform/terraform.tfstate, depending on whether remote state is enabled), one can run terraform remote config to change the remote state configuration.

When this is run, Terraform writes the current local state to the newly-selected remote state storage backend, unconditionally overwriting whatever was there before. This can prove troublesome if, due to user error, the given new configuration points at a location where an unrelated state is already stored: the state is then overwritten, and if that backend doesn't provide any sort of versioning capability (e.g. for Consul) then the state is completely lost.

I would like Terraform to detect this situation and refuse the new remote state configuration. My idea to do this is as follows:

• When a completely new state is created, Terraform generates a UUID and puts it in a new state attribute called "lineage". This value is preserved on all future modifications to the state.
• When terraform remote config is used, Terraform would first read the given new configuration, and:
  • If the read fails, ignore the error and proceed with trying to write.
  • If the read succeeds, unmarshal the retrieved state document and proceed with trying to write only if parsing was successful and the lineage UUID matches the local lineage. Otherwise, fail with an error explaining that there is already something else stored at the given location.

If the user really does want to discard the existing value and write the new one, they must delete the existing object manually using whatever tool is appropriate for the backend in question, after which the initial read will fail and the write will proceed.

Tthere is still a small race condition between Terraform's read and the subsequent write, but at "human scale" this period is small enough that it is very unlikely to cause a problem, and would only be an issue if two users concurrently try to configure Terraform to write a local state to the same location. In that case, the outcome is no worse than in today's implementation.

Does that seem reasonable, or are there edge cases here I'm not considering?

[joonas-fi]

This seems to be implemented? At least there's a lineage field with a UUID in .tfstate.

[apparentlymart]

The Lineage concept was added, but right now it doesn't do anything because it ran into conflict with another change in 0.7: terraform plan no longer persists the refreshed state. Thus the "interesting" part of this change was punted until we could figure out how to make it work again.

The summary of this issue no longer precisely describes the problem but I'm going to leave this here nonetheless as a reminder that I need to revisit this issue at some point.

[apparentlymart]

As of Terraform 0.9, lineage is now checked in various situations. This was improved some more in 0.10 and will be improved further in future, but I'm gonna call this issue done and we'll discuss any further improvements in their own, more-specific issues.

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.