Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(MADR): meshpassthrough design #10253

Merged
merged 11 commits into from
May 22, 2024
137 changes: 137 additions & 0 deletions docs/madr/decisions/050-mesh-passthrough.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# MeshPassthrough

* Status: accepted

## Context and Problem Statement

During the design process of `MeshExternalService`, we discovered that allowing traffic to specific domains and wildcard domains (e.g., `*.aws.us-east-2.com`) is a great functionality. However, this capability doesn't fully fit within the `MeshExternalService` concept. `MeshExternalService` is a kind that allows adding some external services to work within your mesh, be targeted by policies, and be configured by them. Unfortunately, when a user needs to communicate with services that shouldn't be modified by Envoy, we discovered that we need a separate policy to handle them.

## Current solution

Currently, we don't support passthrough mode. Kuma creates DNS entries for real domains and returns a custom IP. This has a limitation: you cannot have two external services using the same domain. Additionally, we don't support wildcard domains.

## Considered Options

* MeshExternalService resource with passthrough mode support
* New MeshPassthrough policy

## Decision Outcome

New `MeshPassthrough` policy seems like cleaner, more flexible.

### Positive Consequences

* More flexibility: We can target specific proxies.
* Clarity: There is no confusion about which fields are involved in the policy configuration.
* Simplicity: Policies are cleaner and cannot be configured with other policies, unlike MeshExternalService.
* No confusion with Universal: There is no issue with non-transparent proxies.
slonka marked this conversation as resolved.
Show resolved Hide resolved

### Negative Consequences

* Additional complexity: Introduction of another policy.
* Resource management: The need to manage two separate resources.

### New MeshPassthrough policy

We would like to introduce a new policy `MeshPassthrough` which allows exposing domains, IPs, CIDRs through specific sidecars.


```yaml
apiVersion: kuma.io/v1alpha1
kind: MeshPassthrough
metadata:
name: custom-passthrough
namespace: kuma-system
labels:
kuma.io/mesh: default
spec:
targetRef:
kind: MeshSubset # Mesh, MeshSubset
tags:
chatgpt.io/access: "true"
default:
appendMatch:
lukidzi marked this conversation as resolved.
Show resolved Hide resolved
- type: Domain
value: api.chatgpt.com
port: 443
protocol: tls
```

`MeshPassthrough` should allow targeting specific subset of proxies and apply configuration only on them. We should support following kinds: `Mesh` and`MeshSubset`.

* **matchAppend**: list of all domains/ips/cidrs supported through the selected sidecars. In case there is many polcies matching the same sidecar, lists are merged.
* **type**: type of the entry, one of `Domain`, `IP` or `CIDR`
* **value**: value for the entry
* **port**: port on which service can communicate
* **protocol**: defines a protocol of the communication. Possible values:
* `Tls`: should be used when TLS traffic is originated by the client application in the case the `kuma.io/protocol` would be tcp
* `Tcp`: WARNING: can't be used when `match.type == Domain` (at TCP level we are not able to disinguish domain, in this case it is going to hijack whole traffic on this port). This will be validated in the config.
* `Grpc`
* `Http`
* `Http2`
lukidzi marked this conversation as resolved.
Show resolved Hide resolved

slonka marked this conversation as resolved.
Show resolved Hide resolved
#### Implementation

We can use [Matcher API](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/matching/matching_api.html#matching-api) which got out of a Alpha phase and is stable. We can use it to support different entries on one listener. Together with [passthrough](https://kuma.io/docs/2.7.x/networking/non-mesh-traffic/#outgoing) mode one the mesh, MeshOperator can disable all outgoing traffic except the one provided by `MeshPassthrough`. It is worth pointing out that when `passthrough.enabled` is set to `true` on the `Mesh`, `MeshPassthrough` policies have no effect because all outgoing traffic is allowed.

From the envoy configuration point of view, we are going to add filter chain matchers under `outbound:passthrough:ipv4"` listener.

lukidzi marked this conversation as resolved.
Show resolved Hide resolved
#### Universal without transparent proxy

This policy won't apply without transparent proxy.
lukidzi marked this conversation as resolved.
Show resolved Hide resolved

#### ZoneEgress

TBA
lukidzi marked this conversation as resolved.
Show resolved Hide resolved

#### Security

It is advised that the MeshOperator is responsible for the `MeshPassthrough` policy. This policy can introduce traffic outside of the mesh or even the cluster, and the MeshOperator should be aware of this. It is also important to note that once a policy is matched by tags, any service owner can add specific tags that match the policy. Depending on the security requirements, the MeshOperator:

* control how service owners use these tags (OPA, Kyverno, MeshConstraints...),
* should conduct audits and proceed based on the audit results.

### Other options

#### MeshExternalService resource with passthrough mode support

As described in `MeshExternalService`(MADR-xxx TBA), we can create a separate `type: Passthrough`.

```yaml
kind: MeshExternalService
metadata:
name: example
namespace: kuma-system
labels:
kuma.io/mesh: default
kuma.io/zone: east-1
spec:
match:
- type: Domain
value: *.aws.us-east-2.com
port: 80
protocol: http
- type: Domain
value: httpbin.com
port: 80
protocol: http
- type: CIDR
value: 10.1.1.0/24
port: 80
protocol: http
- type: IP
value: 192.168.0.1
port: 80
protocol: http
type: Passthrough
```

### Positive Consequences

* One policy handling both cases

### Negative Consequences

* Confusing way of targeting `MeshExternalServices` with policies but not affecting `Passthrough` clusters
* Match with many entries doesn't fit logically with `type: Managed`
* Not possible to apply only on specific sidecar
lukidzi marked this conversation as resolved.
Show resolved Hide resolved
Loading