Cannot create/use repository in S3 with S3 Object Lock enabled

[eriksw]

Output of restic version

restic 0.9.4 compiled with go1.11.4 on linux/amd64

How did you run restic exactly?

RESTIC_REPOSITORY=s3:s3.amazonaws.com/[myBucketName]/[myPrefixInBucket] \
RESTIC_PASSWORD_FILE=/home/me/.resticpassword \
./restic init --json

What backend/server/service did you use to store the repository?

S3, with S3 Object Lock enabled and configured to apply to all objects.

Expected behavior

Restic should have initialized the repository.

Actual behavior

Fatal: create key in repository at s3:s3.amazonaws.com/[myBucketName]/[myPrefixInBucket] failed: client.PutObject: Content-MD5 HTTP header is required for Put Object requests with Object Lock parameters

Steps to reproduce the behavior

Create a bucket with s3 object lock enabled and configured to apply to all objects. Here's some (partial) terraform code:

resource "aws_s3_bucket" "bucket" {
  bucket = "${var.name}"

  versioning {
    enabled = true
  }

  lifecycle_rule {
    enabled = true
    prefix  = ""

    abort_incomplete_multipart_upload_days = "7"

    noncurrent_version_expiration {
      days = "${var.noncurrent_version_expiration_days}"
    }
  }

  object_lock_configuration {
    object_lock_enabled = "Enabled"

    rule {
      default_retention {
        mode = "COMPLIANCE"
        days = "${var.object_lock_days}"
      }
    }
  }
}

variable "name" {
  type    = "string"
  default = "myBucketName"
}

variable "noncurrent_version_expiration_days" {
  type        = "string"
  description = "In reality, this would be large, like 30-90."
  default     = "1"
}

variable "object_lock_days" {
  type        = "string"
  description = "In reality this would be large, like 180. Don't make this big when testing!"
  default     = "3"
}

Do you have any idea what may have caused this?

Restic is attempting to PUT to S3 without computing Content-MD5 and including that in the request.

Do you have an idea how to solve the issue?

Compute Content-MD5 and set the header when making requests to s3.

Did restic help you or made you happy in any way?

😐

[eriksw]

Attempting to use the rclone backend doesn't solve this either:

$ cat ~/.config/rclone/rclone.conf
[s3]
type = s3
env_auth = true
region = us-east-1
storage_class = STANDARD_IA
disable_checksum = false

$ RESTIC_REPOSITORY=rclone:s3:[myBucjetName]/[myPrefixInBucket] \
RESTIC_PASSWORD_FILE=/home/[me]/.resticpassword \
restic init --json
rclone: 2019/03/08 20:01:07 ERROR : keys/[...]: Post request put error: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>
rclone: <Error><Code>InvalidRequest</Code><Message>Content-MD5 HTTP header is required for Put Object requests with Object Lock parameters</Message><RequestId>[...]</RequestId><HostId>[...]</HostId></Error>
rclone: 2019/03/08 20:01:07 ERROR : keys/[...]: Post request rcat error: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>
rclone: <Error><Code>InvalidRequest</Code><Message>Content-MD5 HTTP header is required for Put Object requests with Object Lock parameters</Message><RequestId>[...]</RequestId><HostId>[...]</HostId></Error>
Fatal: create key in repository at rclone:s3:[myBucketName]/[myPrefixInBucket] failed: server response unexpected: 500 Internal Server Error (500)

[ibib]

Restic use the minio library for the S3 backend. As for now it seems that the Object Lock feature ist not implemented.
minio/minio#7002

[eriksw]

@ibib There is no need for Restic to perform any PutObjectLegalHold/PutObjectRetention api calls, so minio/minio#7002 isn't actually the blocker for fixing this.

What is necessary is for the Content-MD5 to be calculated and set on PutObject requests made to S3. It looks like the minio could be lacking in this area (I only skimmed the docs for the minio s3 client), but I do see bits in the minio code that look like it supports it on the (minio) server side.

[ibib]

Oh. I'm sorry. I got the server and the client repository mixed up.

[eriksw]

Ah, I hadn't even realized there was a different repository for the client. :)

[chasebolt]

would like to see this feature as we can't currently use restic with s3 and object locking enabled.

[instantlinux]

Is this on the roadmap? Ransomware has been all over the news for months. The only reliable mechanism supplied by AWS and other S3-compatible cloud providers for defending against an attacker who has obtained API credentials to the targeted account's data is Object Lock. This is a critical feature.

[rawtaz]

Not in particular. There's a lot of features and other things requiring attention. This is just one of them. There are other ways to solve the problem by using snapshots and what not, S3 isn't the only supported storage that you can use. But you probably know that there are alternatives.

[instantlinux]

My point is that the "other ways to solve the problem" do not work in the face of today's ransomware attackers. A leaked AWS API credential can be used by an attacker to defeat any AWS-based alternative. Object Lock is the gold standard for protection against ransomware: please consider prioritizing this above "a lot of features". As for non-AWS solutions, I don't believe there is better protection against ransomware on any other cloud storage service; you'd have to air-gap your backups using Iron Mountain's pickup/drop-off vault services (or the equivalent using your own vehicle).

[rawtaz]

My point is that the "other ways to solve the problem" do not work in the face of today's ransomware attackers.

Why would no other solution work to prevent ransomware attacks? Are you suggesting that the only viable solution to ransomware attacks are object locks?

Regarding AWS, what if the attacker manages to gain access to Amazon's systems and kills the object lock? To completely avoid that, you'd need to air-cap those systems too, just like anything else.

I get you, you want this feature. I'm not disputing that, just pointing out that there are other ways to reach similar protection (all of them have their pros and cons of course).

BTW, there's a similar discussion (and some related references) in #3195.