Add auth server TLS listener and SPIFFE middleware (#6843)

SPIFFE X.509-SVID client authentication needs the client certificate
from a TLS handshake, but the MCP proxy port serves plain HTTP. Give
the embedded authorization server its own TLS listener on port 8443,
so the token endpoint can see client certificates without moving MCP
traffic to HTTPS.

The listener requests client certificates only when spiffe_x509 is
configured, sends the SPIFFE CAs as a hint, and reloads the serving
certificate when its files change, keeping the last good pair.
spiffeauth.Middleware extracts a claimed SPIFFE ID from the presented
peer certificate into the request context; it does not verify the
certificate. Malformed SPIFFE identities receive an OAuth JSON error.
Credential validation remains the responsibility of the client
authentication strategy.

EmbeddedAuthServer gains Start, and the proxy runner starts the
listener on the same host as the MCP port, rejecting a proxy or
diagnostics port that collides with it. Constructors reject configured
TLS listeners without a listener host so unsupported callers fail
explicitly. Graceful shutdown force-closes connections on timeout.

SPIFFE trust configuration remains rejected at startup; this change
does not yet enable SPIFFE client authentication.

Co-authored-by: Mecatl <noreply@mecatl.dev>

Author: jhrozek

docs/arch/09-operator-architecture.md

@@ -238,7 +238,7 @@ MCPExternalAuthConfig resources can be referenced via two paths:
 
 **Controller**: `cmd/thv-operator/controllers/mcpexternalauthconfig_controller.go`
 
-`EmbeddedAuthServerConfig` can carry top-level SPIFFE trust-domain declarations in `spiffeTrustDomains` and canonical client associations in `inboundGrants.spiffeClientAuth` (`cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go`). The operator converts these CRD fields into runtime config via `buildSPIFFETrustDomainRunConfigs`/`buildSPIFFEClientAuthRunConfigs` in `cmd/thv-operator/pkg/controllerutil/authserver.go`, and projects file-source ConfigMap data read-only. `newServer` constructs the configured bundle source and passes its X.509 and JWT views to the authorization server. However, `RunConfig.Validate()` still rejects any non-empty SPIFFE trust configuration before the runner starts, so this wiring is not reachable in normal deployment. TLS listener startup and X.509-SVID verification are not implemented. See [SPIFFE Association Declarations](18-spiffe-association-declarations.md) for current scope.
+`EmbeddedAuthServerConfig` can carry top-level SPIFFE trust-domain declarations in `spiffeTrustDomains` and canonical client associations in `inboundGrants.spiffeClientAuth` (`cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go`). The operator converts these CRD fields into runtime config via `buildSPIFFETrustDomainRunConfigs`/`buildSPIFFEClientAuthRunConfigs` in `cmd/thv-operator/pkg/controllerutil/authserver.go`, and projects file-source ConfigMap data read-only. `newServer` constructs the configured bundle source and passes its X.509 and JWT views to the authorization server. However, `RunConfig.Validate()` still rejects any non-empty SPIFFE trust configuration before the runner starts, so this wiring is not reachable for SPIFFE authentication in normal deployment. The proxy runner can start the auth-server TLS listener on fixed port 8443, but X.509-SVID verification is not implemented. vMCP listener startup and operator configuration or network exposure for this listener are not supported; configuring it through the current vMCP constructor fails because no listener host is supplied. See [SPIFFE Association Declarations](18-spiffe-association-declarations.md) for current scope.
 
 ### MCPOIDCConfig
 

docs/arch/11-auth-server-storage.md

@@ -640,7 +640,7 @@ When the embedded authorization server is deployed in an environment that cannot
 
 **Current status: not reachable in this build.** `RunConfig.Validate()` rejects any non-empty `spiffeTrustDomains`/`inboundGrants.spiffeClientAuth` before the authorization-server runner is created (`validateSPIFFENotYetEnforced` in `pkg/authserver/config.go`; `pkg/authserver/runner/embeddedauthserver.go`), so storage creation and decorator installation never happen. The decorator, overlay, and durable-reservation behavior below is the design this epic has implemented and tested in isolation, not current operational behavior — see [SPIFFE Association Declarations](18-spiffe-association-declarations.md) for the full status.
 
-When top-level `spiffeTrustDomains` and `inboundGrants.spiffeClientAuth` are configured, the embedded authorization server wraps its storage backend in a `SPIFFEStorageDecorator` — installed as the outermost decorator, after CIMD (`decorateStorageForSPIFFE` in `pkg/authserver/server_impl.go`). This decorator overlays statically configured OAuth clients ahead of the dynamic DCR/CIMD backend. `newServer` also wires the configured bundle source into the X.509 and JWT verification interfaces, but normal startup remains blocked by `RunConfig.Validate()`. The decorator reserves configured client IDs; it does not itself verify SVIDs. TLS listener startup and X.509-SVID verification are not implemented.
+When top-level `spiffeTrustDomains` and `inboundGrants.spiffeClientAuth` are configured, the embedded authorization server wraps its storage backend in a `SPIFFEStorageDecorator` — installed as the outermost decorator, after CIMD (`decorateStorageForSPIFFE` in `pkg/authserver/server_impl.go`). This decorator overlays statically configured OAuth clients ahead of the dynamic DCR/CIMD backend. `newServer` also wires the configured bundle source into the X.509 and JWT verification interfaces, but normal SPIFFE startup remains blocked by `RunConfig.Validate()`. The proxy runner can serve authorization-server routes on its TLS listener at fixed port 8443; the listener does not implement X.509-SVID verification, and configured SPIFFE authentication remains unavailable. vMCP does not start this listener, and operator configuration/exposure is not supported. The decorator reserves configured client IDs; it does not itself verify SVIDs.
 
 ### What it does
 

docs/arch/18-spiffe-association-declarations.md

@@ -1,8 +1,24 @@
 # SPIFFE Association Declarations
 
-**Current status: SPIFFE configuration is still rejected at startup.** `RunConfig.Validate()` validates the trust declarations, then `validateSPIFFENotYetEnforced` rejects any non-empty `spiffeTrustDomains` before the runner starts. This branch does wire `newServer` to construct the configured bundle source and pass its X.509 and JWT views into the authorization server, but that wiring is not reachable through normal startup. TLS-listener startup and X.509-SVID verification are not implemented, so SPIFFE client authentication is not deployable yet.
+**Current status: SPIFFE configuration is still rejected at startup.** `RunConfig.Validate()` validates the trust declarations, then `validateSPIFFENotYetEnforced` rejects any non-empty `spiffeTrustDomains` before the runner starts. This branch wires `newServer` to construct the configured bundle source and pass its X.509 and JWT views into the authorization server, and the proxy runner can start the auth-server TLS listener. However, X.509-SVID verification is not implemented and SPIFFE client authentication is not deployable yet.
 
-The embedded authorization server carries the SPIFFE association model across the `RunConfig` boundary, and the file-bundle source can load and poll a local trust-bundle file. The source wiring is present in server construction; the startup validation gate still prevents configured associations from being used. This document describes the implemented components and remaining gaps, not an available deployment capability.
+The embedded authorization server carries the SPIFFE association model across the `RunConfig` boundary, and the file-bundle source can load and poll a local trust-bundle file. The source wiring is present in server construction; the startup validation gate still prevents configured associations from being used. This document describes the implemented components and remaining gaps, not an available SPIFFE deployment capability.
+
+## Auth-server TLS listener
+
+The proxy runner can serve authorization-server routes on an additional TLS listener at fixed port `8443`. Configure the serving certificate and private key in the embedded auth-server `RunConfig`; mount both files read-only into the runner and keep the private key accessible only to the runner:
+
+```yaml
+tls_listener:
+  cert_file: /etc/toolhive/auth-server/tls.crt
+  key_file: /etc/toolhive/auth-server/tls.key
+```
+
+The listener binds the runner's configured host (falling back to loopback when the runner host is unset). A missing or invalid initial certificate/key pair, a bind failure, or a port-8443 conflict with the proxy or enabled diagnostics listener fails startup; the listener does not select another port. Certificate files are checked lazily during TLS handshakes, at most once every 30 seconds. A valid changed pair is used for subsequent handshakes; a failed reload keeps serving the last good pair.
+
+This listener serves OAuth/OIDC authorization-server routes only. Those routes remain mounted on the MCP transport port as well; enabling the listener does not make MCP traffic HTTPS. The listener uses the runner's bind host, so binding a non-loopback address exposes it on that network interface: restrict network reachability with deployment firewall/network policy controls and protect the certificate key. SPIFFE trust-domain configuration is still rejected at startup, and X.509-SVID verification is not implemented, so this listener does not make SPIFFE authentication deployable. vMCP does not start this listener: its current constructor call supplies no listener host, so configuring `tls_listener` there fails clearly at startup. Operator configuration and Service/port or certificate-mount exposure for this listener are not supported.
+
+The token-endpoint middleware rejects certificates containing a malformed SPIFFE identity with HTTP 401 and JSON `{"error":"invalid_client"}`, using `Content-Type: application/json` and `Cache-Control: no-store`. Rejected requests do not reach the authentication strategy; certificates without a SPIFFE URI SAN pass through unchanged. Extracting a claimed SPIFFE ID does not verify the certificate or enable SPIFFE authentication.
 
 ## Model and boundaries
 
@@ -104,11 +120,11 @@ JWT-SVID assertions currently have no application-level replay protection, `jti`
 
 Configuration and loaded bundles are not authentication by themselves. A client ID, a declared association, a request header, an unverified SPIFFE-looking URI, a client-supplied trust domain, or a loaded bundle is never workload identity. JWT-SVID validation establishes identity only after the assertion validates against configured trust material and the association registry authorizes the resulting SPIFFE ID and configured client ID.
 
-The JWT-SVID client-authentication path described above is implemented by [#6203](https://github.com/stacklok/toolhive/issues/6203). `newServer` (`pkg/authserver/server_impl.go`) now constructs and wires the multi-domain bundle source into both JWT and X.509 verification interfaces. However, `RunConfig.Validate()` rejects every non-empty SPIFFE trust configuration before normal server startup, so neither authentication path is currently deployable. The file loader and its rotation logic are wired into server construction, but are not reachable through that startup path. TLS listener startup and X.509-SVID verification ([#6202](https://github.com/stacklok/toolhive/issues/6202)) remain unimplemented; Workload API and bundle-endpoint loaders, SPIFFE method integration with grants or discovery metadata ([#6204](https://github.com/stacklok/toolhive/issues/6204)), and deploying SPIRE or mounting Workload API sockets ([#6205](https://github.com/stacklok/toolhive/issues/6205)) also remain pending.
+The JWT-SVID client-authentication path described above is implemented by [#6203](https://github.com/stacklok/toolhive/issues/6203). `newServer` (`pkg/authserver/server_impl.go`) constructs and wires the multi-domain bundle source into both JWT and X.509 verification interfaces. However, `RunConfig.Validate()` rejects every non-empty SPIFFE trust configuration before normal server startup, so neither authentication path is currently deployable. The file loader and its rotation logic are wired into server construction, but are not reachable through that startup path. The proxy runner's auth-server TLS listener is implemented and documented above; X.509-SVID verification ([#6202](https://github.com/stacklok/toolhive/issues/6202)) remains unimplemented. Workload API and bundle-endpoint loaders, SPIFFE method integration with grants or discovery metadata ([#6204](https://github.com/stacklok/toolhive/issues/6204)), and deploying SPIRE or mounting Workload API sockets ([#6205](https://github.com/stacklok/toolhive/issues/6205)) also remain pending.
 
 For [#6205](https://github.com/stacklok/toolhive/issues/6205), `workloadapi.X509Source` implements both `x509svid.Source` and `x509bundle.Source`, so one Workload API connection can also provide the authorization server's own certificate when deployment wiring is added. The v1alpha1 `ClientCASecretRef` plus `subPath` shape cannot support a rotating bundle and must not be reused for this purpose.
 
-Configured SPIFFE associations remain non-deployable independent of the above: `RunConfig.Validate()` still hard-rejects any non-empty `spiffeTrustDomains` (see "Current status" above), so none of this runs against a real deployment yet.
+Configured SPIFFE associations remain non-deployable: `RunConfig.Validate()` still hard-rejects any non-empty `spiffeTrustDomains` (see "Current status" above), so none of the SPIFFE authentication paths runs against a real deployment yet.
 
 Future X.509-SVID credential-validation code must establish identity from validated SVIDs and then resolve that verified identity through this registry. It must fail closed for missing associations, client-ID ownership mismatches, unknown trust domains, and methods not enabled by policy.
 

docs/arch/README.md

@@ -142,8 +142,9 @@ Welcome to the ToolHive architecture documentation. This directory contains comp
 
 18. **[SPIFFE Association Declarations](18-spiffe-association-declarations.md)**
     - SPIFFE configuration remains rejected at startup; the feature is not yet deployable
-    - File-bundle source and `newServer` X.509/JWT bundle wiring are present but unreachable through normal startup
-    - JWT-SVID validation logic exists; X.509-SVID verification and TLS listener startup remain unimplemented
+    - Auth-server TLS listener runs on the proxy runner at fixed port 8443; vMCP and operator configuration/exposure are unsupported
+    - File-bundle source and `newServer` X.509/JWT bundle wiring are present but unreachable through normal SPIFFE startup
+    - JWT-SVID validation logic exists; X.509-SVID verification remains unimplemented
     - Fail-closed policy validation and durable, restart-safe static-client reservation
     - Endpoint/Workload API loading and SPIFFE method integration with grants or discovery remain pending
 

docs/server/docs.go

@@ -33,7 +33,7 @@ import (
 // CurrentSchemaVersion is the current version of the authserver RunConfig schema.
 const CurrentSchemaVersion = "v0.1.0"
 
-// TLSListenerRunConfig describes the auth server TLS listener configuration.
+// TLSListenerRunConfig configures the auth server TLS listener on port 8443.
 type TLSListenerRunConfig struct {
 	CertFile string `json:"cert_file" yaml:"cert_file"`
 	KeyFile  string `json:"key_file" yaml:"key_file"`
@@ -275,8 +275,8 @@ type RunConfig struct {
 	// non-nil value explicitly controls grant-family enablement.
 	InboundGrants *InboundGrantsRunConfig `json:"inbound_grants,omitempty" yaml:"inbound_grants,omitempty"`
 
-	// TLSListener configures auth server routes to be served over TLS on port 8443.
-	// It is currently rejected because the runner does not start this listener.
+	// TLSListener serves the auth server routes over TLS on port 8443, in
+	// addition to the MCP port. Required for spiffe_x509 client authentication.
 	TLSListener *TLSListenerRunConfig `json:"tls_listener,omitempty" yaml:"tls_listener,omitempty"`
 
 	// RegistrationRateLimit bounds the unauthenticated /oauth/register
@@ -354,11 +354,8 @@ type DelegateClientRunConfig struct {
 // catches operator-supplied misconfiguration early so server startup fails
 // loudly instead of degrading silently at runtime.
 func (c *RunConfig) Validate() error {
-	if c.TLSListener != nil {
-		return errors.New(
-			"tls_listener: the TLS listener is not yet supported by this build; " +
-				"remove this configuration until listener startup is implemented",
-		)
+	if err := c.TLSListener.validate(); err != nil {
+		return err
 	}
 	if err := c.validateClientAdmission(); err != nil {
 		return err
@@ -424,6 +421,25 @@ func (c *RunConfig) validateClientAdmission() error {
 	return nil
 }
 
+// validate requires both certificate paths when a TLS listener is configured.
+// A nil receiver means no listener and is valid.
+func (l *TLSListenerRunConfig) validate() error {
+	if l == nil {
+		return nil
+	}
+	missing := make([]string, 0, 2)
+	if l.CertFile == "" {
+		missing = append(missing, "tls_listener.cert_file")
+	}
+	if l.KeyFile == "" {
+		missing = append(missing, "tls_listener.key_file")
+	}
+	if len(missing) > 0 {
+		return fmt.Errorf("%s must be set", strings.Join(missing, " and "))
+	}
+	return nil
+}
+
 // validateUpstreamFilter rejects an UpstreamFilter configuration whose rules or
 // defaults reference upstream names that are not configured, that reference the
 // mandatory first upstream, or that otherwise violate the filter's static

docs/server/swagger.json

@@ -2039,14 +2039,23 @@ func TestRunConfigValidateTLSListener(t *testing.T) {
 	}{
 		{name: "nil listener"},
 		{
-			name:       "empty listener config is rejected as unsupported",
-			config:     &TLSListenerRunConfig{},
-			wantErrMsg: "tls_listener: the TLS listener is not yet supported",
+			name:   "complete listener config",
+			config: &TLSListenerRunConfig{CertFile: "cert", KeyFile: "key"},
+		},
+		{
+			name:       "missing certificate",
+			config:     &TLSListenerRunConfig{KeyFile: "key"},
+			wantErrMsg: "tls_listener.cert_file",
 		},
 		{
-			name:       "complete listener config is rejected as unsupported",
-			config:     &TLSListenerRunConfig{CertFile: "cert", KeyFile: "key"},
-			wantErrMsg: "tls_listener: the TLS listener is not yet supported",
+			name:       "missing key",
+			config:     &TLSListenerRunConfig{CertFile: "cert"},
+			wantErrMsg: "tls_listener.key_file",
+		},
+		{
+			name:       "missing both",
+			config:     &TLSListenerRunConfig{},
+			wantErrMsg: "tls_listener.cert_file and tls_listener.key_file",
 		},
 	}
 	for _, tt := range tests {