Taxonomy of backup / redundancy / failsafe levels

[mbuechse]

Do we have a taxonomy of failsafe levels?

For instance, the emerging standard on volume types refers to replication, and in this case, it is mostly to protect against a failure of a storage device, so we specify neither the number of replicas nor whether they should span multiple zones etc. In other cases, however, we might want to protect against power loss or fire or other risks.

So it would be interesting to define multiple levels of "failsafe" that may be applied to replication, backups and the like, and to establish handy nomenclature.

[josephineSei]

Use Cases for Redundancy and their probability

Use Case	Probability	Consequences
Disk Failure/Loss	High	Data loss on this disk. Impact depends on type of lost data (data base, user data)
Node Outage	Medium to High	Data loss on node / (temporary) loss of functionality and connectivity of node (impact depends on type of node)
Rack Outage	Medium	similar to Disk Failure and Node Outage
Power Outage (Data Center supply)	Medium	potential data loss, temporary loss of functionality and connectivity of node (impact depends on type of node)
Fire	Medium	permanently Disk and Node loss in the affected zone
Flood	Low	permanently Disk and Node loss in the affected zone
Earthquake	Very Low	permanently Disk and Node loss in the affected zone
Storm/Tornado	Low	permanently Disk and Node loss in the affected fire zone
Cyber threat	High	permanently loss of data on affected Disk and Node

Grouping those use cases

Group	level of affection	Use Cases
1. Level	single volumes, VMs...	Disk Failure, Node outage, (maybe rack outage)
2. Level	number of resources, most of the time recoverable	Rack outage, (Fire), (Power outage when different power supplies exist)
3. Level	lots of resources / user data + potentially not recoverable	Fire, Earthquake, Storm/Tornado, Power Outage
4. Level	complete deployment, not recoverable	Flood, Fire

Redundancy in OpenStack Resources

Replication of Volumes

Replication of Volumes can be achieved when either a backend is used, that already provides replication OR it can be defined in a volume type, which then uses two different (instances of) backends.

The replication provided here can vary from just being replicated on a different disk/ssd in the same rack or up to mirroring to another Rack and depending on the physical infrastructure up to being mirrored into a different fire zone.

Nevertheless volume replication should be seen as the simplest kind of redundancy (Level 1). The one that is used for simple hardware failure such as a disk failure or a node failure.

Replication of Objects (Object Storage)

Nowadays most Installations do not use Swift directly which means dealing with the Swift internal replication (https://docs.openstack.org/swift/latest/overview_replication.html) is not needed.

Instead many deployments use RBD RadosGW with a ceph backend. Here the internal ceph replication can be used.

This results in the replication being mainly used as the simplest kind of redundancy (Level 1). Due to the nature of objects, they can be easily replicated and stored by the users themself. So a user can easily store their object data in different locations which would result in a redundancy of Level 4.

Replication of server data (VMs)

Due to the nature of data in use and their constant changing, it is not easy to provide replication on the IaaS-Layer.

There are two different versions of VMs that needs to be considered:

1. ephemeral storage VMs

2. volume-based VMs

3. The ephemeral storage is stored directly on the compute node. This in combination with the data being in used and updated makes replication impossible. Everything that should be redundant, MUST be handled by the IaaS-User.

4. Volume-based VMs make use of possible replication of the Volumes. As soon as new data is written or updated, the blocks are automatically replicated through the volume storage solution. This means replication of Level 1 would be given. BUT: due to the data being in use and blocks being written instead of files, the consistency of the data is not always given, and there will be a "short" delay while transferring data from the compute node to the storage backend.
   In this case users will always have the need to check for consistency themself and be aware of having maybe outdated (up to a few minutes) data. In case of a node outage (compute host) there is a good chance of being able to reconstruct user data from just a few minutes ago.
   For any higher level of redundancy users SHOULD involve redundancy mechanisms on higher layers than IaaS.

Replication of Secrets

The keys used for volume encryption and stored in Barbican are necessary to decrypt user data. The keys are stored encrypted (a key is encrypted with an project-key-encryption-key and that KEK is again stored encrypted with a master kek) in a simple data base. The Project-KEKs are either stored in a database or in an HSM and the Master-KEK is stored according to the Barbican Plugin. The database is always deployed redundant on different nodes and would survive Level 1 and maybe Level 2 Failure. Barbican should always be deployed redundant too, which makes the Master-KEK the only possible single point of failure. It depends on where the Master KEK is stored (within each Barbican instance or within a Network-HSM-cluster would be failure safe for Level 1 and maybe Level 2): if it is stored in a single HSM a failure of that would render all encrypted data (that is not in use) impossible to access.

The CSP could be able to backup the MasterKEK either through the life-cycle tool or through a dedicated backup. Which would help here.

[josephineSei]

@mbuechse I can go on and cover the other OpenStack resources or you can.

[mbuechse]

@josephineSei Feel free to cover more. I mainly created the issue so we don't forget about the topic. I wanted to bring it up in the IaaS call to collect opinions, but probably not before next week. If you want, you can bring it up even this week.

[anjastrunk]

IMO: This taxonomy influences several standards in IaaS and KaaS, such as availability zones and regions in OpenStack or base security features in Kubernetes. We should create a task force containing of all stakeholders.