Orders fail with `certificate: time limit exceeded` when the upstream CA takes >30s to issue after finalization (CertiNext) — lego `Certificate.Timeout` is never set

[csgriggs]

Summary

When the upstream CA takes longer than 30 seconds to issue a certificate after /finalize, acme-proxy's async finalization fails with certificate: time limit exceeded, and the downstream client's order goes invalid. The CA still completes the order on its side, so every client retry mints another uncollected certificate.

This is the upstream-side sibling of #22. #22 fixed the downstream half (clients now get status: processing + Retry-After instead of a hung connection). The remaining problem is that the proxy's own wait on the CA is capped at lego's default 30s Certificate.Timeout, which createLegoClient in externalcas/external.go never overrides — even though the surrounding request context (RequestTimeout()) already allows 2 minutes.

CertiNext (InCommon Certificate Service) has confirmed to us that their post-finalization checks can take longer than 30s. Since acme-proxy is widely used in front of CertiNext, this likely affects many deployments.

Environment

• acme-proxy 0.5.1 (ghcr.io/esnet/acme-proxy:0.5.1), Docker Compose
• authority.type: externalcas, EAB against https://acme-us.certinext.io/v1/directory
• Also verified the code path is unchanged on v0.7.1 / main

Logs

Two consecutive attempts for the same host. Note the failure lands ~31s after "requesting certificates" both times, and the authorization was already valid (no DCV in the window — this is pure issuance latency):

2026/09/02 04:00:04 [INFO] [<host>.unr.edu] acme: authorization already valid; skipping challenge
2026/09/02 04:00:04 [INFO] [<host>.unr.edu] acme: Validations succeeded; requesting certificates
2026/09/02 04:00:36 ERROR async finalization failed order=ZL35pwYe45UuFtLb3ZCh8Rwb86k2zIql err="error signing certificate for order ZL35pwYe45UuFtLb3ZCh8Rwb86k2zIql: authority.Sign; error creating certificate: failed to obtain certificate: error: one or more domains had a problem:\n<host>.unr.edu: certificate: time limit exceeded"
2026/09/02 04:17:28 INFO processing certificate request domains=[<host>.unr.edu]
2026/09/02 04:17:28 [INFO] [<host>.unr.edu] acme: Obtaining bundled SAN certificate given a CSR
2026/09/02 04:17:29 [INFO] [<host>.unr.edu] AuthURL: https://acme-us.certinext.io/v1/acme/authz/ebf0f915-e11e-4b57-b22a-fc9104f2f1e0
2026/09/02 04:17:29 [INFO] [<host>.unr.edu] acme: authorization already valid; skipping challenge
2026/09/02 04:17:29 [INFO] [<host>.unr.edu] acme: Validations succeeded; requesting certificates
2026/09/02 04:18:00 ERROR async finalization failed order=4hjN6P0d8TXvylAAOgIpK4AsPM6Nb9j4 err="error signing certificate for order 4hjN6P0d8TXvylAAOgIpK4AsPM6Nb9j4: authority.Sign; error creating certificate: failed to obtain certificate: error: one or more domains had a problem:\n<host>.unr.edu: certificate: time limit exceeded"

The startup log's Wait for certificate [timeout: 30s, interval: 500ms] is the same lego wait.

Side effect: orphaned duplicate certificates

Because the CA finishes the order after lego has given up, each retry produces a real, CT-logged certificate that no one collects. For the single host above, crt.sh shows ~50 certificates issued on 2026-09-02 alone. Besides the noise, that risks tripping the CA's duplicate-certificate rate limits, which can make the latency worse.

Root cause

externalcas/external.go, createLegoClient:

clientConfig := lego.NewConfig(user)
clientConfig.CADirURL = cfg.CaURL
clientConfig.Certificate.KeyType = certcrypto.EC256
clientConfig.HTTPClient = &http.Client{Timeout: cfg.HTTPTimeout()}

lego.NewConfig initializes Certificate.Timeout to 30s and nothing changes it, so lego's post-finalize poll for the certificate gives up at 30s regardless of the 2-minute RequestTimeout() context wrapping the request.

Proposed fix

Make the wait configurable, defaulting to the current 30s so nothing changes for existing deployments:

• externalcas/config.go: add cert_obtain_timeout (seconds, omitempty) plus an ObtainTimeout() time.Duration accessor (default 30s), validated to be non-negative and below RequestTimeout().
• externalcas/external.go: clientConfig.Certificate.Timeout = cfg.ObtainTimeout().
• Tests for the default, a configured value, and the validation errors; README row documenting the field.

I have this patched and tested on top of v0.7.1 and am happy to open a PR.

[netops2devops]

Hi @csgriggs!

Thanks for reaching out.

1. Can you please elaborate a bit more about your environment? What does your docker-compose.yml file look like? What about your ca.json file? Did you try running acme-proxy using the installer script? Are you using EAB mode or the new DNS01-TXT mode?

2. Have you tried removing acme-proxy from the path (for testing purposes) and just use LEGO with EAB to get certificates from Certinext? Do you get a similar timeout error when order finalization exceeds 30s with certinext?

3. We upgraded Lego to v5 a couple of weeks ago but the changes are currently sitting in an unmerged branch. Can you please test against this branch https://github.com/esnet/acme-proxy/tree/upgrade/lego-v4-v5 and report back? When you start acme-proxy, try with this environment variable set

$ LEGO_CERT_TIMEOUT=300 /opt/acme-proxy/step-ca ca.json

Cheers!

[csgriggs]

1. Environment. Containers via docker-compose, not the installer. Caddy terminates TLS on the public name and reverse-proxies /acme/standard/* and /acme/ucc/* to two acme-proxy containers (one per Certinext EAB profile) on an internal bridge network. EAB mode. Compose is:

ervices:
  caddy:
    image: caddy:2.11.4
    restart: unless-stopped
    ports: ["80:80", "443:443"]
    volumes:
      - ./caddy/Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
    env_file: .env
    networks: [acme-net]
    depends_on: [acme-proxy-standard, acme-proxy-ucc]

  acme-proxy-standard:
    image: ghcr.io/csgriggs/acme-proxy:0.7.1-unr.2   # v0.7.1 + PR #57
    restart: unless-stopped
    volumes:
      - ./acme-proxy-standard/ca.json:/opt/acme-proxy/ca.json:ro
      - acme_std_db:/opt/acme-proxy/db
    networks: [acme-net]

  acme-proxy-ucc:
    image: ghcr.io/csgriggs/acme-proxy:0.7.1-unr.2
    restart: unless-stopped
    volumes:
      - ./acme-proxy-ucc/ca.json:/opt/acme-proxy/ca.json:ro
      - acme_ucc_db:/opt/acme-proxy/db
    networks: [acme-net]

volumes: { caddy_data: {}, acme_std_db: {}, acme_ucc_db: {} }
networks: { acme-net: { driver: bridge } }

Sanitized ca.json (the ucc one differs only in provisioner name and EAB pair):

{
  "address": ":443",
  "dnsNames": ["<internal proxy name>"],
  "db": { "type": "bbolt", "dataSource": "/opt/acme-proxy/db/bbolt" },
  "authority": {
    "type": "externalcas",
    "config": {
      "ca_url": "https://acme-us.certinext.io/v1/directory",
      "account_email": "<ops mailbox>",
      "eab_kid": "<redacted>",
      "eab_hmac_key": "<redacted>",
      "cert_obtain_timeout": 300,
      "request_timeout": 330
    },
    "provisioners": [{ "type": "ACME", "name": "standard",
      "claims": { "enableSSHCA": false, "disableRenewal": false, "allowRenewalAfterExpiry": false, "disableSmallstepExtensions": true } }],
    "backdate": "1m0s"
  }
}

2. Plain lego against Certinext. Yes. We also run lego (v4.18) directly with EAB against acme-us.certinext.io on another system and saw the identical failure: the default 30 s wait expires with time limit exceeded while Certinext issues the certificate anyway. --cert.timeout 300 fixed it there (mid-August). That's what prompted externalcas: make the certificate obtain timeout configurable #57 — acme-proxy had no equivalent knob. Certinext support has since confirmed the delay is on their side (a CT log operator slow for the 30-day product) and recommended a 300 s finalization wait.

3. Will do. One thing from reading the branch first: LEGO_CERT_TIMEOUT is the lego CLI's variable — the library's lego.NewConfig still defaults Certificate.Timeout to 30 s in v5.4.0, and I don't see the branch reading that env var or setting the field, so I expect the same 30 s unless it's set. I'll test both as-is and with externalcas: make the certificate obtain timeout configurable #57 rebased onto the v5 branch, and post the numbers.

[csgriggs]

Results, as promised.

upgrade/lego-v4-v5 on our sandbox (Certinext sandbox CA, EAB mode, acme.sh client):

Build | Startup log | Issuance -- | -- | -- branch as-is, LEGO_CERT_TIMEOUT=300 in the environment | msg="Waiting for certificates." timeout=30s interval=500ms | PASS, 25 s end to end branch + the two commits from #57 | msg="Waiting for certificates." timeout=5m0s interval=5s | PASS, 23 s end to end

So the v5 branch runs fine, but LEGO_CERT_TIMEOUT is the lego CLI's variable — the library keeps NewConfig's 30 s default unless Certificate.Timeout is set. #57 cherry-picks onto the v5 branch with one trivial conflict (the removed Certificate.KeyType line); happy to rebase the PR onto that branch if you'd prefer it land there.

A testing caveat worth knowing: Certinext's sandbox issues from a staging chain that isn't publicly trusted, so nothing is CT-logged and the slow-CT-operator delay never occurs there. Only production reproduces it — which is why the sandbox runs above are fast under both builds.

Production, lego v4.33 + cert_obtain_timeout: 300, ~20 h: 9 finalize waits — 8 succeeded (22, 23, 32, 40, 43, 50, 88 and 140 s; six of them beyond the stock 30 s, one beyond 90 s), and 1 exceeded 300 s and failed. After that failure the client retried with the same CSR and Certinext answered badCSR: You are attempting to submit a CSR of an existing certificate order — the timed-out order had issued anyway. Two conclusions: the wait needs to be operator-configurable rather than any fixed value, and a timed-out order should be assumed issued at the CA.