Add support for certificates from Let's Encrypt.

This adds a new configuration option named `acme` and a new sub-command
`montagu renew-certificate`. Using these will fetch a certificate from
Let's Encrypt (or another ACME provider), inject it into the proxy
container and finally reload nginx. The host machine needs to be
configured to run the `renew-certificate` on a regular basis, using, for
example, crontab or systemd timers.

The ACME protocol is handled by running certbot as its own short-lived
container. Alternatives could have been a long-running container running
a cron-like daemon, installing certbot inside the nginx container and
using `docker exec` to run it, installing and running certbot directly
on the host, using a Python implementation of the ACME protocol directly
inside the montagu tool process.

The certbot container shares a volume with nginx to host the
acme-challenge files on port 80. To allow nginx to start and serve these
files before we have any certificate at all, the proxy generates a
self-signed certificate on startup if none exist yet. This certificate
would be replaced on the first renewal.

See vimc/montagu-proxy#87 for the proxy side of this, mapping the
`/.well-known/acme-challenge` path to a directory on the file system.

While we support alternative ACME servers, in production we will
probably stick to the default which uses Let's Encrypt. The
configuration option makes it easier to trial the process using servers
that aren't exposed publically, using a miniature ACME server such as
[pebble][pebble]

In the future the ACME configuration block may be extended to support
DNS challenges instead, allowing us to use this even on internal-only
services. If and when this happens, we won't even need nginx to host
acme-challenges and may be able to remove the need for the transient
self-signed certificates.

Once we get more experience with this, we may also want to move some of
this functionality into constellation, allowing it to be re-used in
other projects.

[pebble]: https://github.com/letsencrypt/pebble

config/complete/montagu.yml

@@ -94,7 +94,6 @@ proxy:
   ssl:
     key: "k3y"
     certificate: "cert"
-    dhparam: "param"
 contrib:
   name: montagu-contrib-portal
   tag: master

src/montagu_deploy/cli.py

@@ -4,6 +4,7 @@
   montagu status <path>
   montagu stop <path> [--volumes] [--network] [--kill] [--force]
     [--extra=PATH] [--option=OPTION]...
+  montagu renew-certificate <path> [--force-renewal]
 
 Options:
   --extra=PATH     Path, relative to <path>, of yml file of additional
@@ -19,14 +20,17 @@
                    signal their running configuration, or if config cannot be
                    parsed. Use with extra and/or option to force stop with
                    configuration options.
+  --force-renewal  Renew the certificate, even if the current one isn't close to
+                   expiry.
 """
 
 import docopt
 import yaml
 
 import montagu_deploy.__about__ as about
 from montagu_deploy.config import MontaguConfig
-from montagu_deploy.montagu_constellation import MontaguConstellation
+from montagu_deploy.montagu_constellation import montagu_constellation, proxy_update_certificate
+from montagu_deploy.certbot import obtain_certificate
 
 
 def main(argv=None):
@@ -37,11 +41,13 @@ def main(argv=None):
         cfg = MontaguConfig(path, extra, options)
         obj = montagu_constellation(cfg)
         if args.action == "start":
-            montagu_start(obj, args)
+            montagu_start(obj, args, cfg)
         elif args.action == "status":
             montagu_status(obj)
         elif args.action == "stop":
             montagu_stop(obj, args, cfg)
+        elif args.action == "renew-certificate":
+            montagu_renew_certificate(obj, cfg, force_renewal=args.force_renewal)
         return True
 
 
@@ -53,14 +59,29 @@ def parse_args(argv=None):
     return path, extra, options, MontaguArgs(opts)
 
 
-def montagu_start(obj, args):
+def montagu_start(obj, args, cfg):
     obj.start(pull_images=args.pull)
 
+    if cfg.ssl_mode == "acme":
+        montagu_renew_certificate(obj, cfg)
+
 
 def montagu_status(obj):
     obj.status()
 
 
+def montagu_renew_certificate(obj, cfg, *, force_renewal=False):
+    if cfg.ssl_mode != "acme":
+        msg = "Proxy is not configured to use automatic certificates"
+        raise Exception(msg)
+
+    print("Renewing certificates")
+    (cert, key) = obtain_certificate(obj, cfg, force_renewal=force_renewal)
+
+    container = obj.containers.get("proxy", cfg.container_prefix)
+    proxy_update_certificate(container, cert, key, reload=True)
+
+
 def montagu_stop(obj, args, cfg):
     if args.volumes:
         verify_data_loss(cfg)
@@ -123,9 +144,12 @@ def __init__(self, args):
             self.action = "status"
         elif args["stop"]:
             self.action = "stop"
+        elif args["renew-certificate"]:
+            self.action = "renew-certificate"
 
         self.pull = args["--pull"]
         self.kill = args["--kill"]
         self.volumes = args["--volumes"]
         self.network = args["--network"]
         self.version = args["--version"]
+        self.force_renewal = args["--force-renewal"]

src/montagu_deploy/config.py

@@ -13,14 +13,7 @@ def __init__(self, path, extra=None, options=None):
         self.vault = config.config_vault(dat, ["vault"])
         self.network = config.config_string(dat, ["network"])
         self.protect_data = config.config_boolean(dat, ["protect_data"])
-        self.volumes = {
-            "db": config.config_string(dat, ["volumes", "db"]),
-            "emails": config.config_string(dat, ["volumes", "emails"]),
-            "burden_estimates": config.config_string(dat, ["volumes", "burden_estimates"]),
-            "templates": config.config_string(dat, ["volumes", "templates"]),
-            "guidance": config.config_string(dat, ["volumes", "guidance"]),
-            "mq": config.config_string(dat, ["volumes", "mq"]),
-        }
+        self.volumes = config.config_dict(dat, ["volumes"])
 
         self.container_prefix = config.config_string(dat, ["container_prefix"])
         self.repo = config.config_string(dat, ["repo"])
@@ -58,15 +51,25 @@ def __init__(self, path, extra=None, options=None):
 
         # Proxy
         self.proxy_ref = self.build_ref(dat, "proxy")
-        self.proxy_ssl_self_signed = "ssl" not in dat["proxy"]
-        if not self.proxy_ssl_self_signed:
-            self.ssl_certificate = config.config_string(dat, ["proxy", "ssl", "certificate"])
-            self.ssl_key = config.config_string(dat, ["proxy", "ssl", "key"])
-            self.dhparam = config.config_string(dat, ["proxy", "ssl", "dhparam"])
         self.proxy_port_http = config.config_integer(dat, ["proxy", "port_http"])
         self.proxy_port_https = config.config_integer(dat, ["proxy", "port_https"])
         self.proxy_metrics_ref = self.build_ref(dat["proxy"], "metrics")
 
+        if "ssl" in dat["proxy"] and "acme" in dat["proxy"]:
+            msg = "Cannot specify both ssl and acme options in proxy options."
+            raise Exception(msg)
+        if "ssl" in dat["proxy"]:
+            self.ssl_mode = "static"
+            self.ssl_certificate = config.config_string(dat, ["proxy", "ssl", "certificate"])
+            self.ssl_key = config.config_string(dat, ["proxy", "ssl", "key"])
+        elif "acme" in dat["proxy"]:
+            self.ssl_mode = "acme"
+            self.acme_email = config.config_string(dat, ["proxy", "acme", "email"])
+            self.acme_server = config.config_string(dat, ["proxy", "acme", "server"], is_optional=True)
+            self.acme_no_verify_ssl = config.config_boolean(dat, ["proxy", "acme", "no_verify_ssl"], is_optional=True)
+        else:
+            self.ssl_mode = "self-signed"
+
         # Portals
         self.admin_ref = self.build_ref(dat, "admin")
         self.contrib_ref = self.build_ref(dat, "contrib")

src/montagu_deploy/montagu_constellation.py

@@ -209,30 +209,43 @@ def inject_api_config(container, cfg):
 def proxy_container(cfg):
     name = cfg.containers["proxy"]
     proxy_ports = [cfg.proxy_port_http, cfg.proxy_port_https]
+
+    mounts = []
+
+    if cfg.ssl_mode == "acme":
+        mounts.extend([
+            constellation.ConstellationMount("acme-challenge", "/var/www/.well-known/acme-challenge", read_only=True),
+            constellation.ConstellationMount("certificates", "/etc/montagu/proxy"),
+        ])
+
     return constellation.ConstellationContainer(
         name,
         cfg.proxy_ref,
         ports=proxy_ports,
         args=[str(cfg.proxy_port_https), cfg.hostname],
-        configure=proxy_configure,
+        preconfigure=proxy_preconfigure,
+        mounts=mounts,
     )
 
 
-def proxy_configure(container, cfg):
-    print("[proxy] Configuring reverse proxy")
+def proxy_update_certificate(container, cert, key, *, reload):
+    print("[proxy] Copying ssl certificate and key into proxy")
     ssl_path = "/etc/montagu/proxy"
-    if cfg.proxy_ssl_self_signed:
-        print("[proxy] Generating self-signed certificates for proxy")
-        docker_util.exec_safely(container, ["self-signed-certificate", ssl_path])
-    else:
-        print("[proxy] Copying ssl certificate and key into proxy")
-        docker_util.exec_safely(container, f"mkdir -p {ssl_path}")
-        docker_util.string_into_container(cfg.ssl_certificate, container, join(ssl_path, "certificate.pem"))
-        docker_util.string_into_container(cfg.ssl_key, container, join(ssl_path, "ssl_key.pem"))
-        docker_util.string_into_container(cfg.dhparam, container, join(ssl_path, "dhparam.pem"))
+    docker_util.string_into_container(cert, container, join(ssl_path, "certificate.pem"))
+    docker_util.string_into_container(key, container, join(ssl_path, "ssl_key.pem"))
 
+    if reload:
+        print("[proxy] Reloading nginx")
+        docker_util.exec_safely(container, f"nginx -s reload")
 
 
+def proxy_preconfigure(container, cfg):
+    # In self-signed mode, the container generates its own certificate on its
+    # own. Similarly, in ACME mode, the container generates its own certificate
+    # and after starting we request a new one.
+    if cfg.ssl_mode == 'static':
+        print("[proxy] Configuring reverse proxy")
+        proxy_update_certificate(container, cfg.ssl_certificate, cfg.ssl_key, reload=False)
 
 
 def proxy_metrics_container(cfg):