tls: Use ACMEv2 and support automatic wildcard certificates

[mholt]

1. What does this change do, exactly?

Adds ACMEv2 support, which includes fully-managed, automatic wildcard certificates.

It also adds {labelN} placeholders, and support for distributed auto-HTTPS of the HTTP challenge.

2. Please link to the relevant issues.

Closes #2067

3. Which documentation changes (if any) need to be made because of this PR?

• How to use wildcard certificates:

*.example.com
tls {
    dns <provider>
}

In other words: use a proper wildcard pattern for the site name, and enable the DNS challenge. For a wildcard certificate, only one wildcard label is allowed, and it must be the left-most label of the domain.

Wildcards can also be forced, but this should only be done when there are many subdomains configured differently that would otherwise trigger rate limits with the CA:

sub1.example.com {
    tls {
        dns <provider>
        wildcard
    }
}
.
.
.
sub150000.example.com {
    tls {
        dns <provider>
        wildcard
    }
}

• Document recommended use case for wildcard certificates and gently discourage other tempting ones.

• Clarify allowed usage of * in hostnames on the HTTP Caddyfile page. These haven't changed, but should be clarified, since what is allowed by Caddy is slightly more liberal than what is allowed for wildcard certificates (Caddy allows a superset of wildcard patterns to that of what certs allow).

• Document {labelN} placeholders, which get replaced with the values of specific labels in the hostname ({label1} matches sub in sub.example.com or whatever the value is in the place of the wildcard in *.example.com)

• Update the conditions for automatic HTTPS on the Automatic HTTPS page (a specific wildcard pattern is now permitted with automatic HTTPS)

• Explain how to use Caddy behind a load balancer when using automatic HTTPS: the only requirement is that the $CADDYPATH/acme folder should be shared (usually by mounting locally). And the obvious requirement that the LB redirect port 80 traffic to a Caddy instance serving HTTP. The Caddy instances will then synchronize the management of the certificates and share them, rather than each instance doing its own thing.

4. Checklist

• I have written tests and verified that they fail without my change
• I have squashed any insignificant commits
• This change has comments for package types, values, functions, and non-obvious lines of code
• I am willing to help maintain this change if there are issues with it later

5. TODO

• Wait for the acmev2 xenolf/lego to be finished and moved into the existing acme folder, then update our vendoring and import paths (EDIT: We will not wait for a release of lego, it's working fine now in a branch, and we vendor the dependency and commit it, anyway)
• Before merging into master, replace the hard-coded staging v2 endpoint as default with the production v2 endpoint.

In the announcement blog post, we will need to help site owners understand the move to ACME v2. Basically, it will act as if you were running Caddy for the first time: an email address will be needed, and all new certificates will be obtained. This is because the format of the metadata in ACME v2 has changed from v1. This was deemed the simplest way to do it. Overall, we expect a very smooth transition for nearly everyone.

Please test this out! Now is the time to catch bugs.

[lbguilherme]

I have an use case that doesn't seem to be covered, which is having several (30~40) subdomains, each with its own different caddy configuration. Currently caddy very often fail to add a new domain because of let's encrypt rate limits (20 issues per base domain per week).

It would be very very good if caddy could (either by configuration or automatically) try to get a wildcard certificate to share between many subdomain configurations.

Is this feasible at all? (I understand that this is very probably subject of future work and wouldn't go with this PR)

The other option for us is to run one caddy with wildcard certs as reverse proxy of another internal caddy that only use http.

[lbguilherme]

[...] and all new certificates will be obtained. This is because the format of the metadata in ACME v2 has changed from v1. This was deemed the simplest way to do it. Overall, we expect a very smooth transition for nearly everyone.

Basically this will break caddy for me, because I have more than 20 subdomains and lets encrypt won't issue that many certificates

[mholt]

@lbguilherme

It would be very very good if caddy could (either by configuration or automatically) try to get a wildcard certificate to share between many subdomain configurations.

It definitely won't happen automatically, because the added risk of a wildcard credential should never be assumed for the user.

How about a wildcard subdirective to the tls subdirective that attempts to get a wildcard certificate instead of a normal one. Its use would be discouraged in the docs except for situations like yours, which IMO is one of the only justifiable reasons to use a wildcard certificate.

So:

tls {
    dns <provider>
    wildcard
}

?

[lbguilherme]

Seems a perfect solution. Just one question:

my.domain.with.many.labels.example.com {
  tls {
    dns <provider>
    wildcard
  }
}

Would it try to issue *.domain.with.many.labels.example.com or *.with.many.labels.example.com or ... or *.example.com?

[mholt]

@lbguilherme I've pushed a commit that adds a wildcard subdirective. Will you please test it out ASAP? This branch already defaults to the ACMEv2 staging endpoint so it won't count against your rate limit or anything.

[mholt]

Gah, GitHub didn't update the page with your new comment.

In that example you gave, it will obtain the only allowable wildcard for that hostname: *.domain.with.many.labels.example.com

[lbguilherme]

<3

I gave it a try and it works except for when On-Demand TLS is also enabled:

tls {
    protocols tls1.0 tls1.2
    key_type rsa4096
    dns cloudflare
    max_certs 99999
    wildcard
}

This issues a non-wildcard certificate. Without max_certs 99999 is works like a charm. Maybe still reading from cfg.Addr.Host somewhere?

[mholt]

@lbguilherme Thanks for trying it!!

I gave it a try and it works except for when On-Demand TLS is also enabled. ... Maybe still reading from cfg.Addr.Host somewhere?

No, on-demand TLS uses the SNI value, which is why it didn't issue a wildcard certificate. Hmmm.... I can't off the top of my head think of a reason why you would need to issue a wildcard certificate on-demand. Why can't it just be obtained regularly, without on-demand mode?

[lbguilherme]

I enabled on-demand because without it adding a new domain with flawed configuration (or when hitting lets encrypt rate limits) it would take the whole server down (it won't start). With on-demand everywhere we only fail the minimum.

Now that we use SIGUSR1, maybe this decision can be revisited.

[mholt]

@lbguilherme

would take the whole server down (it won't start).

Well, it can't take a server down that wasn't running. :) Yes, as you said, using SIGUSR1 is the best way to reload configuration.

And now, LE rate limits shouldn't be a problem anyway.

I'm going to leave it as-is and see how this fares over the course of the next release. Thanks for your feedback!

[francislavoie]

LTGM! I'm not the most proficient at Go, but all the ideas all seem good. Been thoroughly discussed in Slack 👍