SPICE-0022: Custom HTTP Headers

spices/SPICE-0022-http-headers.adoc

@@ -0,0 +1,167 @@
+= Custom HTTP Headers
+
+* Proposal: link:./SPICE-0022-http-headers.adoc[SPICE-0022]
+* Author: link:https://github.com/kyokuping[kyokuping]
+* Status: Proposed
+* Category: Language, Standard Library, Tooling
+
+== Introduction
+
+This SPICE proposes a new evaluator option to allow adding custom HTTP headers to outbound requests, with the ability to target specific URL prefixes.
+
+== Motivation
+
+When Pkl interacts with HTTP resources, it may need to provide authentication tokens or other custom headers to access them. For example, a Pkl module might be hosted in a private repository that requires an `Authorization` header for access.
+
+Currently, there is no way to add custom headers to Pkl's HTTP requests. This limits Pkl's ability to interact with a wide range of HTTP-based resources.
+
+== Proposed Solution
+
+A new HTTP setting will be added, called `headers`. This setting allows users to specify a list of headers to be added to outbound HTTP requests. To avoid leaking credentials, headers can be configured on a per-URL-pattern basis.
+
+For example, to provide an authentication token for specific subdomains, the configuration would look like this:
+
+.~/.pkl/settings.pkl
+[source,pkl]
+----
+amends "pkl:settings"
+
+http {
+  headers {
+    ["https://*.my.private.repo/*"] {
+      ["authorization"] = "Bearer my-secret-token"
+    }
+  }
+}
+----
+
+This will add the `authorization` header to all requests sent to `https://*.my.private.repo/` (subdomains of `my.private.repo`) and its subpaths.
+
+== Detailed design
+
+The `pkl.EvaluatorSettings.Http` class will be extended with a new `headers` property.
+
+.pkl:EvaluatorSettings
+[source,pkl]
+----
+class Http {
+  // ... existing properties
+
+  /// HTTP headers to add to outbound requests targeting specified URLs.
+  headers: Mapping<URLPattern, Mapping<HttpHeaderName, *Listing<String>|String>>?
+}
+----
+
+The details for `URLPattern` and `HttpHeaderName` will be provided in the following sections.
+
+A new CLI flag, `--http-header`, will be introduced to allow specifying headers from the command line. The flag will accept key-value pairs in the format `<url-pattern>=<header-name>:<header-value>[,<header-name>:<header-value>...]`.
+
+Example:
+[source,shell]
+----
+pkl eval \
+  --http-header "https://*.my.private.repo/*:authorization=Bearer my-secret-token" \
+  myModule.pkl
+----
+
+The HTTP client will be modified to check for matching header configurations for each outgoing request and add the corresponding headers.
+
+=== The `URLPattern` typealias
+
+The `URLPattern` typealias defines glob patterns that are used to match the URLs of HTTP requests and determine which HTTP headers to apply.
+
+To prevent accidental matches, the pattern should end with a slash (`/`) or a wildcard (`*`).
+
+If several patterns match a given request URL, the one with the highest specificity is selected. The specificity of a pattern is evaluated using the following process:
+
+1. Decompose the pattern into its components (scheme, host, path).
+2. Compute a score for the pattern by combining the host score and the path score:
+  - First, calculate a "base score" by adding points for each character and glob feature in the string:
+    - 10 points for every literal character (including escaped characters)
+    - 5 points for each character set (`[]`)
+    - 4 points for every single character wildcard (`?`)
+    - 2 points for each wildcard (`*`)
+    - 1 point for every globstar (`**`)
+    - For alternations (`{}`), use the average score of the included patterns.
+  - Then, multiply the host score by a weight of 100.
+3. When two patterns have identical scores, the longer pattern is considered more specific.
+
+=== The `HttpHeaderName` typealias
+
+The `HttpHeaderName` typealias is used to represent the name of an HTTP header, defined as follows:
+
+.pkl:EvaluatorSettings
+[source,pkl]
+----
+// Reserved HTTP header names that are inappropriate to be set by the user
+typealias ReservedHttpHeaderName =
+  "accept-charset"
+  | "accept-encoding"
+  | "access-control-request-headers"
+  | "access-control-request-method"
+  | "connection"
+  | "content-length"
+  | "cookie"
+  | "date"
+  | "dnt"
+  | "expect"
+  | "host"
+  | "keep-alive"
+  | "origin"
+  | "permissions-policy"
+  | "referer"
+  | "te"
+  | "trailer"
+  | "transfer-encoding"
+  | "upgrade"
+  | "via"
+
+// Reserved HTTP header prefixes that are inappropriate to be set by the user
+const local ReservedHttpHeaderPrefix = new Listing {
+  "proxy-"
+  "sec-"
+  "access-control-"
+}
+const local hasReservedHttpHeaderPrefix = (header: String) ->
+  ReservedHttpHeaderPrefix.any((it) -> header.startsWith(it))
+
+// Regex for validating HTTP header names based on RFC 7230
+const local httpHeaderNameRegex = Regex("^[a-zA-Z0-9!#\\$%&'*+-.^_`|~]+$")
+const local hasValidHttpHeaderName = (header: String) ->
+    httpHeaderNameRegex.findMatchesIn(header)
+
+typealias HttpHeaderName = String(
+  this == toLowerCase(), // Only accept lowercase header names
+  !(this is ReservedHttpHeaderName),
+  !hasReservedHttpHeaderPrefix.apply(this),
+  hasValidHttpHeaderName
+)
+----
+
+== Compatibility
+
+This change is strictly backward-compatible. Existing Pkl programs will continue to work as-is.
+
+== Future directions
+
+Future enhancements could include support for more complex header manipulation, such as removing default headers.
+
+== Alternatives considered
+
+=== Global HTTP headers
+
+The initial proposal involved a simpler approach of having a single set of custom headers that would be applied to all outbound HTTP requests. This was rejected due to security concerns. Sending the same set of headers, which might include authentication tokens, to all hosts is a significant security risk. The per-URL-prefix approach provides a more secure way to manage custom headers.
+
+=== Auth-specific support
+
+Someone could argue it'd make more sense to specifically support auth handling rather than exposing the full ability to add custom HTTP headers. While this can be considered more future-proof in terms of exposing minimal API surface to maintain compatibility, this would limit the potential of supporting many different use cases outside of auth. That being said, adding separate auth support can also be considered later, mostly for supporting complex auth flows like OAuth.
+
+=== Plain Text Patterns
+
+The initial proposal used plain text prefixes for matching URLs. After considering use cases that require more flexible matching, the design was updated to use glob patterns.
+
+Glob patterns allow for more powerful and concise rules. For example, a single pattern like `*.service.internal` can apply a common authentication header to all internal services. Similarly, a pattern like `*.my-company.com` can apply a rule to all subdomains of a company.
+
+== Acknowledgements
+
+Thanks to the Pkl team for their feedback on the initial proposal, which helped shape the more secure and flexible design presented in this SPICE.