From dd6eebefc5186bda86464c8bf4a002fbd2717158 Mon Sep 17 00:00:00 2001 From: Mathew Sims <25371401+mathewcsims@users.noreply.github.com> Date: Thu, 6 Aug 2026 21:42:37 +0100 Subject: [PATCH] Fix tailnet access to LAN-gated apps, and Caddy access logging MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two faults found while debugging docs.mathewcsims.uk being unreachable from an Android client on mobile data. Neither was in this repo's config, and both had been latent for a while. The tailnet fault: a subnet route is a separate ACL destination from the tailnet IPs of the devices behind it. Babel advertised 10.0.1.0/24, the route was approved, and the client had "use subnet routes" on — all three necessary, none sufficient, because no grant named the CIDR. Traffic was dropped by policy at the receiving node, which is why every server-side check came back clean: Caddy, DNS records, the NextDNS rewrite, the tailnet's own resolver and HedgeDoc were all correct throughout. The symptoms actively misdirect. With an exit node the connection times out, because Tailscale keeps RFC1918 destinations off the exit path, so the packet leaves via the local interface and dies. Without one it is refused, because the browser falls back to the public A record and hits handle { abort } from the internet. Neither leaves a trace on the Pi — a full packet capture of the client showed no connection attempt toward the LAN address at all. Adds scripts/tailscale-acl.sh to manage the policy file, in the same shape as dns-nextdns.sh: the API key is used only inside a Python process via urllib so it never appears in argv, /acl/validate runs before anything is applied, and If-Match makes a concurrent console edit a 412 rather than a silent clobber. The logging fault: Caddy emits access logs only for site blocks carrying the log directive. The global log block configures the default logger's sink, not access logging, so no request has ever been logged here. That also means the caddy-abuse fail2ban jail has been watching a file it was never fed — its lifetime counter stood at exactly 3, matching the one synthetic and two hand-fed lines used to verify it nine days ago, and nothing since. Fixed with an (access_log) snippet imported by all 25 site blocks; re-verified that a probe with a unique path now lands in the log and that fail2ban-regex matches the emitted format. SETUP.md's "Accessing LAN-only apps over Tailscale" section said two things were required and claimed the arrangement was confirmed working end to end. It now documents three, and records that no LAN-gated app had ever actually been reachable from a genuinely off-LAN tailnet device — testing from a device on the physical LAN passes regardless, which is how it went unnoticed. Co-Authored-By: Claude Opus 5 --- SETUP.md | 76 ++++++++++++++++++++++--- pi-reverse-proxy/Caddyfile | 46 +++++++++++++++ scripts/tailscale-acl.sh | 112 +++++++++++++++++++++++++++++++++++++ 3 files changed, 227 insertions(+), 7 deletions(-) create mode 100755 scripts/tailscale-acl.sh diff --git a/SETUP.md b/SETUP.md index 1b86ab8..9b87d15 100644 --- a/SETUP.md +++ b/SETUP.md @@ -582,10 +582,12 @@ resolved the hostname. The Pi runs Tailscale, configured as both a subnet router (advertising the LAN) and an exit node — meaning devices elsewhere can reach this network through it, including while off any physical LAN entirely. Every LAN-gated -app in this repo (`mc37`, `apprise`, `vikunja-relay`, `backup`) -needs two separate things to actually be reachable this way, and both were -missing until this was diagnosed directly (bug report → root-caused → fixed -→ verified working, not assumed): +app in this repo (`mc37`, `apprise`, `vikunja-relay`, `backup`, `author`, +`paperless`, `fj`, `healthlog`, `docs`) +needs **three** separate things to actually be reachable this way. Each is +necessary and none is sufficient, which is what makes this so awkward to +debug: with any one missing, every check you can run on the server comes +back clean. 1. **Caddy has to trust the connection's source IP.** Every LAN-gated block uses `@lan remote_ip private_ranges 100.64.0.0/10` — the appended CIDR is @@ -609,9 +611,57 @@ missing until this was diagnosed directly (bug report → root-caused → fixed subdomains (rather than a per-domain split-DNS rule), this works correctly without any further Tailscale-side configuration. -With both of these true, a LAN-gated app is reachable over Tailscale exactly -as if you were on the physical LAN — confirmed working end-to-end, not just -theorized. +3. **The tailnet ACL has to grant the LAN subnet as a destination, by CIDR.** + This is the one that is easiest to miss, because three *other* things look + like they cover it and none of them does. The Pi advertising + `10.0.1.0/24`, that route being approved in the admin console, and the + client having "use subnet routes" enabled are all necessary — and all + three can be true while the traffic is still dropped. A subnet route is a + **separate ACL destination** from the tailnet IPs of the devices; a grant + like `tag:personal → tag:personal` covers `100.x` addresses only and says + nothing about the LAN behind a subnet router. The policy file needs an + explicit grant: + + ```json + { "src": ["tag:personal"], "dst": ["10.0.1.0/24"], "ip": ["*"] } + ``` + + Manage the policy with `./scripts/tailscale-acl.sh get|put` (validates + before applying, and sends `If-Match` so a concurrent console edit is a + 412 rather than a silent clobber). + + To check this directly rather than guessing, dump the Pi's own view of + the enforced filter — it is the receiving node that drops the packets: + + ```bash + ssh mathew@babel 'sudo tailscale debug netmap' | grep -A2 '"Net": "10\.' + ``` + + If `10.0.1.0/24` is absent, this is the fault. The symptom without it is + confusing enough to be worth recording: with an exit node the connection + *times out* (Tailscale deliberately keeps RFC1918 destinations off the + exit path, so the packet leaves via the local interface and dies), and + without one it is *refused* (the browser falls back to the public A + record, reaches Caddy from the internet, and hits `handle { abort }`). + Neither symptom points at an ACL, and neither leaves any trace on the Pi + — a full `tcpdump` of the client's traffic shows no connection attempt + toward the LAN address at all. + + `tcpdump` is **not** installed on the Pi (`sudo apt-get install -y + tcpdump` if needed, and remove it again afterwards). Worth knowing before + you write a capture command and get a silent `timeout` failure with an + empty output file, as happened here. When capturing, watch the interface + column: tailnet traffic arrives on `tailscale0` from a `100.x` source, + while anything arriving on `eth0` came in off the internet — that + distinction is what identifies this class of fault. + +Diagnosed 2026-08-06, when `docs.mathewcsims.uk` was unreachable from an +Android client on mobile data. Items 1 and 2 had been in place for months +and item 3 never had been — so despite this section previously claiming the +arrangement was "confirmed working end-to-end", no LAN-gated app had ever +actually been reachable from a genuinely off-LAN tailnet device. Anything +tested from a device sitting on the physical LAN at the time would have +passed regardless, which is how it went unnoticed. --- @@ -6953,6 +7003,18 @@ one-off manual commands: from a test IP through the actual monitored log file triggered a real ban (confirmed in both `fail2ban-client status caddy-abuse` and `iptables -L f2b-caddy-abuse`), then cleanly unbanned. + - **Correction (2026-08-06): the global `log {}` block above did not + actually produce access logs**, and this jail was therefore watching a + file that only ever received error-level entries. Caddy's global `log` + option configures the **default logger's sink** — where log lines are + written — but requests are logged only for site blocks carrying the + `log` **directive**, and none had it. Every verification quoted above + still holds, because all three test lines were fed into the file by + hand; that is exactly why it went unnoticed, and why the jail's lifetime + counter stood at precisely 3 (one synthetic + two fed) nine days later. + Fixed by adding an `(access_log)` snippet imported by all 25 site + blocks. Re-verified afterwards: a probe with a unique path now appears + in `logs/access.log`, and `fail2ban-regex` matches the emitted format. **Manual action items — out of scope for this repo, worth doing separately:** diff --git a/pi-reverse-proxy/Caddyfile b/pi-reverse-proxy/Caddyfile index 280e494..f6cb346 100644 --- a/pi-reverse-proxy/Caddyfile +++ b/pi-reverse-proxy/Caddyfile @@ -41,6 +41,27 @@ } } +# Access logging. Caddy does NOT emit access logs merely because the global +# `log` block above exists — that configures the DEFAULT logger's SINK (the +# file it writes to) and nothing more. Requests are logged only for site +# blocks that carry the `log` directive, so every block below imports this. +# +# Verified, not assumed: before this snippet existed, a probe request with a +# unique path appeared in neither logs/access.log nor `docker logs caddy`, +# and the only `request`-bearing lines in the file were error-level entries +# (which carry a request object too, and are easy to mistake for access +# logs). The consequence was that the caddy-abuse fail2ban jail +# (../pi-fail2ban/), whose failregex matches +# `"request":{"remote_ip":""..."uri":"..."}` against this exact file, +# was being fed almost nothing — its lifetime counter stood at 3, i.e. the +# stray error-level line, rather than the request stream it was written to +# watch. Found 2026-08-06 while debugging tailnet access. After this change, +# `fail2ban-regex` matches the emitted format (checked against a synthetic +# /wp-login.php line: 1 matched, 0 missed). +(access_log) { + log +} + # General-purpose per-site, per-IP request budget — coarse abuse/DoS # protection, not a substitute for each app's own auth-specific limits # (copyparty has --ban-pw/--ban-403 built in and enabled by default; Vikunja @@ -99,6 +120,7 @@ # correct https links and logs real client IPs (it trusts the Pi's private # LAN IP by default). {$CP_DOMAIN} { + import access_log import security_headers import general_ratelimit cp reverse_proxy http://{$MAC_IP}:3923 @@ -119,6 +141,7 @@ # used here instead of a second hardcoded IP, for a single source of truth if # the Mac's LAN IP ever changes. prospect-ukri-tus.mathewcsims.uk { + import access_log import security_headers import general_ratelimit memos @@ -178,6 +201,7 @@ prospect-ukri-tus.mathewcsims.uk { # rate-limit zone is needed — no signup zone, # unlike prospect-ukri-tus.mathewcsims.uk above. owl.mathewcsims.uk { + import access_log # Wrapped in an explicit route{} block: without it, Caddyfile's adapter # reorders top-level directives by its own fixed precedence rather than # source order (confirmed live via `caddy adapt`). @@ -273,6 +297,7 @@ owl.mathewcsims.uk { # sharing off, trusted-proxy config so it can tell real clients from the # podman gateway) — see vikunja/compose.yaml for the reasoning. vikunja.mathewcsims.uk { + import access_log import security_headers import general_ratelimit vikunja reverse_proxy http://{$MAC_IP}:3456 @@ -285,6 +310,7 @@ vikunja.mathewcsims.uk { # creates an infinite HTTPS redirect loop — Caddy's reverse_proxy sets this # by default, so nothing extra is needed here for that specifically. blog.mathewcsims.uk { + import access_log import security_headers # Ghost admin's own "View site" panel (/ghost/#/site) embeds the live # site in a same-origin iframe — security_headers' blanket DENY blocks @@ -337,6 +363,7 @@ blog.mathewcsims.uk { # zone needed (nothing here to brute-force or abuse), but the shared # security-headers snippet costs nothing to keep on. ways-of-working.mathewcsims.uk { + import access_log import security_headers redir https://blog.mathewcsims.uk/ways-of-working/ permanent } @@ -350,6 +377,7 @@ ways-of-working.mathewcsims.uk { # — nothing to brute-force or abuse — but it's cheap to keep on for # consistency with every other site. mathewcsims.uk { + import access_log import security_headers import general_ratelimit landing reverse_proxy http://{$MAC_IP}:3080 @@ -362,6 +390,7 @@ mathewcsims.uk { # proxies to plain-HTTP Karakeep. Registration is closed # (DISABLE_SIGNUPS=true in ../karakeep/compose.yaml) now that this is public. karakeep.mathewcsims.uk { + import access_log import security_headers import general_ratelimit karakeep @@ -390,6 +419,7 @@ karakeep.mathewcsims.uk { # own README) — this LAN-gate is its only access control, since there's no # in-app auth option to layer on top of instead. See ../apprise/compose.yaml. apprise.mathewcsims.uk { + import access_log import security_headers import general_ratelimit apprise @@ -413,6 +443,7 @@ apprise.mathewcsims.uk { # (see every other app above) doesn't apply here — 2FA in Kuma's own Settings # is the compensating control instead. See ../uptime-kuma/compose.yaml. status.mathewcsims.uk { + import access_log import security_headers import general_ratelimit status reverse_proxy uptime-kuma:3001 @@ -427,6 +458,7 @@ status.mathewcsims.uk { # relay; the general per-IP rate limit covers brute-force. Caddy upgrades # WebSocket automatically (subscriber connections) — nothing extra needed. ntfy.mathewcsims.uk { + import access_log import security_headers import general_ratelimit ntfy reverse_proxy ntfy:80 @@ -440,6 +472,7 @@ ntfy.mathewcsims.uk { # inside the relay itself (Tailscale-Webhook-Signature header) is the real # access control — see ../tailscale-webhook-relay/compose.yaml. tailscale-relay.mathewcsims.uk { + import access_log import security_headers import general_ratelimit tailscale_relay reverse_proxy tailscale-webhook-relay:8080 @@ -455,6 +488,7 @@ tailscale-relay.mathewcsims.uk { # not this LAN-gate — but every other machine-to-machine app in this repo # gets one anyway, so this does too, for consistency and defense in depth. vikunja-relay.mathewcsims.uk { + import access_log import security_headers import general_ratelimit vikunja_relay @@ -484,6 +518,7 @@ vikunja-relay.mathewcsims.uk { # a self-signed cert over the trusted `pi-shared` network, same reasoning # as the mc37 router-admin block below. backup.mathewcsims.uk { + import access_log import security_headers import general_ratelimit backup @@ -510,6 +545,7 @@ backup.mathewcsims.uk { # "LAN DNS" entry (mc37.mathewcsims.uk → 10.0.1.19) so LAN devices reach the Pi # directly and their source IP is private (passing the check below). mc37.mathewcsims.uk { + import access_log @lan remote_ip private_ranges 100.64.0.0/10 # + Tailscale CGNAT range handle @lan { reverse_proxy https://10.0.1.1:8443 { @@ -539,6 +575,7 @@ mc37.mathewcsims.uk { # neither, so this one targets its actual login POST directly, same # defense-in-depth reasoning as Karakeep's NextAuth zone. author.mathewcsims.uk { + import access_log import security_headers @lan remote_ip private_ranges 100.64.0.0/10 # + Tailscale CGNAT range @@ -585,6 +622,7 @@ author.mathewcsims.uk { # so a mismatch surfaces as a 400 on every form submission, not as a # routing error. paperless.mathewcsims.uk { + import access_log import security_headers @lan remote_ip private_ranges 100.64.0.0/10 # + Tailscale CGNAT range @@ -625,6 +663,7 @@ paperless.mathewcsims.uk { # dedicated tighter zone for it rather than assuming the app already # covers it. fj.mathewcsims.uk { + import access_log import security_headers @lan remote_ip private_ranges 100.64.0.0/10 # + Tailscale CGNAT range @@ -668,6 +707,7 @@ fj.mathewcsims.uk { # control from this point on is: registration closed + passkey-only login # (no password fallback configured) + the rate limit below. healthlog.mathewcsims.uk { + import access_log import security_headers import general_ratelimit healthlog @@ -710,6 +750,7 @@ healthlog.mathewcsims.uk { # just trusted from the UI toggle. Real, permanent access control from here # on: registration closed + the rate limit below. wanderer.mathewcsims.uk { + import access_log import security_headers import general_ratelimit wanderer @@ -749,6 +790,7 @@ wanderer.mathewcsims.uk { # is deliberately NOT used here — it's reserved for a possible future # public gallery-sharing tool, which would need the opposite posture. immich.mathewcsims.uk { + import access_log import security_headers @lan remote_ip private_ranges 100.64.0.0/10 # + Tailscale CGNAT range @@ -797,6 +839,7 @@ immich.mathewcsims.uk { # prefix), so it doesn't catch /admin/manage/ (the actual management UI) # or real short-link slugs. msims.link { + import access_log import security_headers @bare_root path / @@ -850,6 +893,7 @@ msims.link { # (The Etherpad instance this replaces needed SAMEORIGIN for exactly that # reason — a useful contrast, not a precedent to copy.) docs.mathewcsims.uk { + import access_log import security_headers @lan remote_ip private_ranges 100.64.0.0/10 # + Tailscale CGNAT range @@ -897,10 +941,12 @@ docs.mathewcsims.uk { # SNI/Host — including a bare-IP scan of the public address — gets a closed # connection, revealing nothing. :80 { + import access_log abort } :443 { + import access_log tls internal abort } diff --git a/scripts/tailscale-acl.sh b/scripts/tailscale-acl.sh new file mode 100755 index 0000000..63b4a31 --- /dev/null +++ b/scripts/tailscale-acl.sh @@ -0,0 +1,112 @@ +#!/bin/sh +# Read and update the Tailscale tailnet policy file (ACL grants, SSH rules, +# tagOwners) via the API, in the same shape as ./dns-nextdns.sh. +# +# The API key is used only inside a Python process via urllib — never passed +# to curl or any other subprocess, so it never appears in argv. Tailscale +# authenticates with HTTP Basic using the key as the username and an empty +# password, so the header is built by hand here rather than shelling out. +# +# `put` always POSTs to /acl/validate first and refuses to apply a policy the +# API rejects — a malformed policy is applied atomically and can lock every +# device out of every resource, so a syntax error should fail here rather +# than in production. It also sends `If-Match` with the ETag read moments +# earlier, so a concurrent edit in the admin console is a 412 rather than a +# silent clobber. +# +# The policy is HuJSON (JSON plus comments and trailing commas). Round-trip +# it as bytes — do NOT parse and re-serialise it, or every comment in the +# file is destroyed. +# +# Usage: +# ./scripts/tailscale-acl.sh get [outfile] # default: stdout +# ./scripts/tailscale-acl.sh put +set -eu + +ACTION="${1:?Usage: $0 get [outfile] | put }" + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(dirname "$SCRIPT_DIR")" + +export PROTON_PASS_SESSION_DIR="${PROTON_PASS_SESSION_DIR:-/tmp/pass-agent-selfhosted}" +mkdir -p "$PROTON_PASS_SESSION_DIR" + +if ! pass-cli info >/dev/null 2>&1; then + if [ ! -f "$REPO_ROOT/.env" ]; then + echo "No active pass-cli session, and no $REPO_ROOT/.env to auto-login with." >&2 + exit 1 + fi + set -a + . "$REPO_ROOT/.env" + set +a + export PROTON_PASS_PERSONAL_ACCESS_TOKEN="$SECRET_ACCESS_TOKEN" + pass-cli login >/dev/null + unset PROTON_PASS_PERSONAL_ACCESS_TOKEN SECRET_ACCESS_TOKEN +fi + +PROTON_PASS_AGENT_REASON="Tailscale policy file: $ACTION $*" \ + pass-cli item view --vault-name "Self-Hosted Secrets" --item-title "Tailscale" --output json \ + | ACTION="$ACTION" FILE_ARG="${2:-}" python3 -c ' +import base64, json, os, sys, urllib.request, urllib.error + +d = json.load(sys.stdin) +content = d["item"]["content"] +key = None +fields = [f for s in content["content"]["Custom"]["sections"] for f in s["section_fields"]] +# `pass-cli item update --field x=y` writes into a separate top-level +# `extra_fields` array, not into any section — see dns-nextdns.sh. +fields += content.get("extra_fields", []) +for f in fields: + if f["name"] == "TAILSCALE_API_KEY": + key = list(f["content"].values())[0] +if not key: + sys.exit("No TAILSCALE_API_KEY field on the \"Tailscale\" item.") + +AUTH = "Basic " + base64.b64encode(f"{key}:".encode()).decode() +BASE = "https://api.tailscale.com/api/v2/tailnet/-" + + +def call(method, path, body=None, ctype="application/hujson", extra=None): + headers = {"Authorization": AUTH, "Accept": "application/hujson"} + if body is not None: + headers["Content-Type"] = ctype + headers.update(extra or {}) + req = urllib.request.Request(BASE + path, data=body, method=method, headers=headers) + try: + with urllib.request.urlopen(req) as resp: + return resp.read(), dict(resp.headers) + except urllib.error.HTTPError as e: + sys.exit(f"{method} {path} -> HTTP {e.code}: {e.read().decode()}") + + +action = os.environ["ACTION"] +arg = os.environ.get("FILE_ARG") or "" + +if action == "get": + raw, _ = call("GET", "/acl") + if arg: + open(arg, "wb").write(raw) + print(f"Wrote {len(raw)} bytes to {arg}") + else: + sys.stdout.write(raw.decode()) + +elif action == "put": + if not arg: + sys.exit("Usage: tailscale-acl.sh put ") + new = open(arg, "rb").read() + + # Refuse to apply anything the API itself will not accept. + call("POST", "/acl/validate", new) + print("validate: OK") + + _, hdrs = call("GET", "/acl") + etag = hdrs.get("ETag") or hdrs.get("Etag") + if not etag: + sys.exit("No ETag returned; refusing to apply without concurrency protection.") + + call("POST", "/acl", new, extra={"If-Match": etag}) + print(f"applied: {len(new)} bytes (If-Match {etag})") + +else: + sys.exit(f"Unknown action: {action}") +'