feat(uploads): ticket-gated and URL uploads without base64 in the model context

[gutencoder]

Problem

upload-file and upload-voucher-file take the file as base64 inline in the JSON-RPC
body. That makes the file's bytes part of the conversation:

• every byte is billed as input tokens, at roughly a 4:3 base64 expansion on top;
• the encoded blob sits in the transcript, so the contents of a receipt are visible in
  the conversation history;
• the ceiling is the JSON_BODY_LIMIT of 12 MB (src/server.ts), which a ~8 MB source
  file already brushes against once base64-expanded.

All of that to move bytes the model never needs to look at. The model needs the resulting
file id; it does not need the PDF.

Solution

Three new tools in the drafts tier, plus the endpoints they depend on. The base64
tools are untouched and stay available — this is purely additive.

• create-upload-ticket — issues a short-lived single-use ticket and returns both a
  browser URL (drag-and-drop page) and a ready-to-run curl command. Bytes go
  client → server → Lexware; the model only ever sees the file id.
• get-upload-result — reads back the file id once the transfer happened.
• upload-file-from-url — fetches the file server-side from a URL.

Supporting pieces: an in-memory ticket store (src/uploads/tickets.ts), the
GET/POST /upload/:ticket endpoints and their dependency-free page
(src/uploads/routes.ts, page.ts), and an SSRF-hardened URL fetcher
(src/uploads/fetch-url.ts).

Two details worth calling out because they were both found the hard way:

• Body parsing. /upload paths must be deferred from the pre-applied global
  express.json() alongside /mcp. The upload routes read the raw body themselves; if
  the JSON parser runs first, a JSON-content-typed upload arrives as a parsed object
  instead of a Buffer and silently became an empty file. The refactor of the existing
  swap into src/server-body-parsing.ts is what makes that expressible; there is a
  Buffer.isBuffer guard in the route as a second line of defence.
• Filenames. filename travels as X-Filename-B64 (base64url of the UTF-8 bytes),
  never as a raw header value. A header value is Latin-1 on the wire, and fetch()
  throws a TypeError before sending for any character above U+00FF — so an ordinary
  German filename with an en dash or typographic quotes broke the entire upload, and a
  raw-byte header turned Rechnung Müller.pdf into Rechnung Müller.pdf in the books.

Why the upload endpoints sit outside the OAuth gate — deliberately

/upload/:ticket is mounted on server.express, i.e. not behind the /mcp bearer
gate. That is the central design decision in this PR and it is intentional, so please
review it on its merits rather than as an oversight:

1. The uploader is not the MCP client. The whole point is that the bytes come from
   somewhere the MCP client isn't: a browser tab the user opens, or a curl on the
   machine where the file actually lives. Neither holds an MCP access token, and neither
   can obtain one — the OAuth flow issues tokens to the registered MCP client, not to an
   arbitrary browser or shell. Putting the endpoint behind the same gate would make the
   feature unimplementable.

2. The ticket is the credential. It is 24 bytes from randomBytes rendered
   base64url (192 bits), so it is not guessable, and it can only be obtained by calling
   create-upload-ticket — a drafts-tier tool reachable only through the authenticated
   /mcp endpoint. Possession of an unguessable, authenticated-issued, short-lived
   token is the same capability-URL model as a pre-signed upload URL.

3. The capability is narrow and bounded. A ticket permits exactly one action: POST
   one file, of at most 20 MB, into the Lexware file store under the type that was
   fixed when the ticket was issued. It grants no read access, no enumeration, and no
   second use. It expires after 15 minutes.

4. Reuse is blocked, not merely discouraged. claim() sets an inFlight flag
   synchronously and holds it across the async body read and Lexware call, so a retried
   request, a duplicated proxy call or a leaked ticket cannot file a second voucher. A
   ticket that is unknown, expired, in flight or already completed gets a flat 410.
   Failures release the lock through a single catch so a ticket is never stranded.

The honest trade-off: a ticket travels in a URL, and URLs leak — browser history,
shoulder-surfing, a copy-pasted link. The mitigations are the short TTL, single use, and
the narrowness of what the capability permits (write-only, one file, pre-fixed type).
If you would rather this were gated differently — a separate shared secret on the
endpoint, a configurable TTL, or the endpoints disabled unless explicitly enabled — I am
happy to change it; it is a small change and I would rather match your preference than
argue for mine.

The URL allowlist is configurable

upload-file-from-url only fetches from hosts on an allow-list, matched on a dot
boundary so evilsharepoint.com cannot pass as sharepoint.com. The list is set with
LEXWARE_UPLOAD_ALLOWED_HOSTS (comma-separated).

Three properties worth reviewing explicitly:

• The default is the built-in Microsoft file-sharing list
  (sharepoint.com, onedrive.live.com, 1drv.ms, graph.microsoft.com), so leaving
  the variable unset behaves exactly as if it did not exist.
• Setting it replaces the defaults, it does not extend them. Extending would make
  Microsoft's domains impossible to opt out of, which is the wrong default for a
  self-hosted server that may have nothing to do with M365.
• An empty value blocks every host, disabling the tool. An allow-list that cannot be
  emptied cannot be used to switch the feature off, and "empty means allow everything"
  would turn a typo into an open SSRF surface. fetchRemoteFile uses ?? rather than
  || for exactly this reason, and there is a test pinning it.

Naming: LEXWARE_* rather than OAUTH_*/MCP_*. In this project OAUTH_* is strictly
the IdP/token layer and MCP_* is how /mcp itself is protected; server behaviour and
capability settings are LEXWARE_* (LEXWARE_READ_ONLY, LEXWARE_ENABLE_DRAFTS, …).
The ALLOWED_<thing> noun follows the existing OAUTH_ALLOWED_EMAIL_DOMAINS.

The SSRF hardening is independent of the allow-list and applies regardless: DNS
resolution is re-checked against loopback, link-local, private, CGNAT, benchmarking and
multicast ranges, and redirects are re-validated rather than followed blindly.

Limitation: the ticket store is in-process

Stating this plainly rather than leaving it to be discovered, because the README
documents Cloud Run and autoscaling is the normal case there:

• The store is a Map in the server process. A restart drops open tickets. They then
  answer 410 — ticket unknown or expired, which is a clear error rather than a hang, and
  the client can simply ask for a new ticket.
• With more than one instance and no sticky sessions, an upload can land on a different
  instance than the one that issued the ticket, and gets the same 410. In that
  topology the feature is unreliable as built.

Why it is still built this way: the alternative is an external store (Redis, a database,
a bucket), which would be the first infrastructure dependency this server has — today it
is a single container with an API key. That is a real cost to impose on every operator to
serve the multi-instance case. The 15-minute TTL bounds the exposure, and the failure mode
is a clean 410 rather than data loss or a wrong voucher.

If you would rather have it pluggable, the store is already behind a small class
(TicketStore: create / peek / claim / release / complete), so a second
implementation would not disturb the routes. Happy to add the seam — I did not want to
design an extension point into your codebase uninvited.

What changes for existing users

Nothing changes for anyone not using the new tools. upload-file and
upload-voucher-file are unmodified. The three new tools are in the drafts tier, so
LEXWARE_READ_ONLY=true or LEXWARE_ENABLE_DRAFTS=false excludes them exactly like the
other write tools. The /upload/:ticket routes are registered unconditionally, but
without a ticket every request to them is a 410.

LEXWARE_UPLOAD_ALLOWED_HOSTS is new but defaults to the same host list the fetcher
would use without it, so an existing deployment sees no difference.

One behavioural note: the body-parsing deferral now also covers /upload. For /mcp and
all other paths the behaviour is unchanged.

Tests

npm test passes (241 tests / 15 files, up from 135 / 10). The new suites cover the
ticket store lifecycle (issue, claim, single-use, expiry, in-flight lock), the routes
(body-parser ordering, rejection before the body is read, empty body releasing the
ticket, clean JSON errors with no stack traces, case-insensitive routing, filename
sanitization, the full non-ASCII filename matrix), the SSRF guards (each blocked range,
redirect re-validation, Content-Disposition parsing including the RFC 5987
charset'language'value form), and the emitted curl command (single replacement point,
no command substitution, quote-injection resistance — one test asks a real shell to
confirm the quoting property).

For the allow-list specifically: the default applies with the variable unset; a
configured list replaces the defaults (a Microsoft host is then rejected); an empty value
blocks everything at both the config and the fetchRemoteFile layer; and the dot-boundary
property (evilsharepoint.com must not match sharepoint.com, nor
sharepoint.com.evil.com) is covered, along with case-insensitivity and a trailing
root-label dot.

npm run build and docker build pass. npm audit --omit=dev --audit-level=high is
unchanged from main (no dependency changes; the branch adds no runtime dependency).

Docs

CHANGELOG.md under [Unreleased] (including the in-process ticket-store limitation);
README.md tool count 60 → 63, the Drafts tier row describes the ticket route, and the
config table plus .env.example document LEXWARE_UPLOAD_ALLOWED_HOSTS.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a ticket-gated upload flow and SSRF-hardened URL uploads so files can reach Lexware without sending raw bytes through the model context.

Changes:

• Introduces /upload/:ticket GET/POST routes with an in-memory TicketStore, plus a self-contained browser upload page.
• Adds MCP tools for ticket issuance/result polling and for URL-based uploads with host allowlisting + address-blocking.
• Updates body parsing to defer global JSON parsing for /upload (and makes path checks case-insensitive), plus extensive tests and documentation updates.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/uploads-tools.test.ts	Adds unit tests for ticket response + curl command construction (headers, quoting, base64url filename).
tests/uploads-tickets.test.ts	Adds coverage for TicketStore lifecycle (create/claim/release/complete/peek, expiry).
tests/uploads-routes.test.ts	Adds end-to-end tests for /upload/:ticket behavior, body parsing deferral, error mapping, and filename encoding.
tests/uploads-fetch-url.test.ts	Adds tests for SSRF protections: allowed-host matching, blocked IP ranges, redirects, limits, timeout, content-disposition parsing.
tests/tools.test.ts	Updates tool-registration test to include new upload tools + new registerTools signature.
tests/server-body-parsing.test.ts	Adds regression tests for case-insensitive /mcp and /upload path detection.
tests/config.test.ts	Adds tests for LEXWARE_UPLOAD_ALLOWED_HOSTS defaulting/replacement/empty semantics.
src/uploads/tickets.ts	Implements in-memory ticket store with single-use + in-flight locking + expiry.
src/uploads/routes.ts	Implements /upload/:ticket endpoints, raw-body handling, error mapping, X-Filename-B64 decoding, header normalization.
src/uploads/page.ts	Adds dependency-free browser upload page and exported filename base64url encoder source for tests.
src/uploads/fetch-url.ts	Adds SSRF-hardened remote fetching: https-only, host allowlist, DNS resolution + blocked-address checks, redirect revalidation, size/time limits, filename parsing.
src/tools/uploads.ts	Adds upload-related MCP tools and curl command builder; integrates default allowlist.
src/tools/index.ts	Extends registerTools to include upload tools with shared ticket store + base URL.
src/server.ts	Wires body-parsing deferral for /mcp and /upload, mounts upload routes, creates shared ticket store, passes new args to registerTools.
src/server-body-parsing.ts	Extracts and generalizes global-json-parser deferral + adds case-insensitive path match helpers.
src/config.ts	Adds uploadAllowedHosts to config and parsing for LEXWARE_UPLOAD_ALLOWED_HOSTS.
README.md	Documents new tools and LEXWARE_UPLOAD_ALLOWED_HOSTS; updates tool count.
CHANGELOG.md	Documents new ticket-based upload flow, URL upload tool, SSRF hardening, and /upload body parsing deferral.
.env.example	Documents LEXWARE_UPLOAD_ALLOWED_HOSTS usage and defaults.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gutencoder]

Short note to tie these three together, since they landed at once and one of them is
large. No rush on any of it — and if the answer to some or all is "no thanks", that is
a perfectly fine outcome.

One practical thing you may not have noticed: the CI runs on all three are sitting at
action_required. That is GitHub's default for a first-time contributor from a fork —
they need your approval before they start, so the checks look empty rather than failing.

In the meantime I ran what CONTRIBUTING.md lists, on each branch:

	npm run build	npm test	docker build
#32 OAUTH_AUDIENCE	pass	139	pass
#33 OAUTH_SCOPES_SUPPORTED	pass	141	pass
#34 uploads	pass	255	pass

npm audit --omit=dev --audit-level=high exits 1 on all three — but identically so on a
clean main: vite sits in dependencies, so --omit=dev does not filter it. No branch
touches package.json or the lockfile.

On ordering: #32 and #33 both append an OAUTH_* field in the same places, so
whichever you take first, the other needs a trivial rebase. Say the word and I'll do it.

On #34 specifically: it is by far the largest and the most opinionated of the three,
and it puts an endpoint outside the /mcp bearer gate. The PR body argues why, but I am
aware that is a design decision you may simply not want in this project, and a large
unsolicited PR is a lot to land on someone. If the shape is wrong for you, I would rather
change it than have you work around it: a separate shared secret on the endpoint, a
config flag that leaves the routes unregistered by default, a different TTL — whatever
fits. And if it is simply out of scope, say so and I will drop it without hard feelings;
the other two stand on their own.

Finally: on #32 I left two of the Copilot threads open rather than resolving them,
because I disagreed and would rather you had the last word than have it look settled.
Both are one-liners if you prefer the suggestions.

[marselsel]

Thanks for this one too — and particularly for naming the trade-offs yourself rather than leaving them to be found. The section on why /upload/:ticket sits outside the OAuth gate, and the one on the in-process ticket store, are exactly the two things I'd have asked about, and having your reasoning up front means we can discuss the decision instead of discovering it.

I want to set expectations honestly: this one is going to take me longer than #32 and #33. Those were ~95 and ~140 lines against code paths I already knew. This is ~3,300 lines, six new modules, and a new unauthenticated HTTP surface — and it's on a server that fronts real accounting data. I'd rather review it properly than merge it quickly.

I'm merging #32 and #33 first. Before I get into this one in depth, two questions, because the answers change how much of the rest matters:

1. Can the upload endpoints be opt-in? You offered this ("the endpoints disabled unless explicitly enabled") and I think I'd take it. My reasoning: the capability-URL model you describe is sound, and the ticket really is a credential — but it changes this server from "one gate, everything behind it" to "two gates with different properties". For an operator who never uses the feature, an env flag means the surface simply isn't there. Something like LEXWARE_ENABLE_UPLOAD_TICKETS=false by default. Does that break any use case you have in mind?

2. How much do you need upload-file-from-url? It's the piece carrying the most inherent risk — a server-side fetcher is an SSRF surface no matter how carefully it's written, and yours is written carefully. But create-upload-ticket already solves the stated problem (keeping bytes out of the model context), and it does it without the server making outbound requests to attacker-influenceable hosts. If the ticket flow covers your actual workflow, splitting the URL fetcher into its own PR would let the ticket half land much sooner.

On the in-process store: for my deployment it's a non-issue (--max-instances=1), so I'm not going to ask you to build a Redis dependency. I'd rather take the TicketStore seam you mention and document the constraint loudly — which your CHANGELOG entry already does.

Give me a bit of time on the full read-through. The volume of tests here is reassuring, but I want to actually trace the SSRF guards and the ticket lifecycle myself rather than trust the test names.

[marselsel]

Merged — thank you! The ticket flow from this PR is now on main via #38 and released as v0.1.11: create-upload-ticket / get-upload-result, the single-use /upload/:ticket route, the drag-and-drop page, the X-Filename-B64 design and the body-parsing deferral. Your two commits were cherry-picked onto current main exactly as you authored them — not squashed — so the history credits you directly.

upload-file-from-url I've deferred rather than rejected, and you deserve the honest reasoning: a server-side fetcher is SSRF surface by construction, and while your implementation held up well under review (the allow-list-first design, per-hop re-validation, and the fail-closed address parser all survived adversarial checking), there is a DNS-rebinding TOCTOU between the lookup() check and the connection fetch() makes with its own resolution. Your comments call this out and argue the allow-list makes it moot — which is true for the built-in Microsoft domains — but it goes live the moment an operator sets a custom LEXWARE_UPLOAD_ALLOWED_HOSTS, and "safe unless you touch the knob" isn't a property I want to ship silently on a server that fronts accounting data. Your branch is preserved; if you'd enjoy taking a crack at connection-level IP pinning (a custom undici dispatcher that connects to the vetted address), I'd gladly review that as its own PR, shipped disabled-by-default.

Review of the ticket flow surfaced a few things now fixed on top of your commits (details in #38): curl's --data-binary silently declaring application/x-www-form-urlencoded whenever no Content-Type was baked into the command — your fallback-chain design was right, curl was quietly defeating it; the upload result expiring on the ticket's creation clock, so a minute-14 upload left get-upload-result a sub-minute read window; and a per-ticket bound on concurrent body buffering for the window before claim() runs.

The measured, reproduce-first writeups in your commits and PR description made this genuinely enjoyable to review — that's now three solid contributions (#32, #33, and this). Closing as superseded by #38.