From 471df45b415dafb256df23d4f75d4541794b03de Mon Sep 17 00:00:00 2001 From: Pau Capdevila Date: Wed, 5 Mar 2025 23:53:01 +0100 Subject: [PATCH 1/6] (architecture): add Fabric Management page Signed-off-by: Pau Capdevila --- docs/architecture/.pages | 1 + docs/architecture/management.md | 122 ++++++++++++++++++++++++++++++++ 2 files changed, 123 insertions(+) create mode 100644 docs/architecture/management.md diff --git a/docs/architecture/.pages b/docs/architecture/.pages index 64933a12..3ab22a1e 100644 --- a/docs/architecture/.pages +++ b/docs/architecture/.pages @@ -1,4 +1,5 @@ nav: - Overview: overview.md + - Fabric Management: management.md - Fabric Implementation: fabric.md - ... diff --git a/docs/architecture/management.md b/docs/architecture/management.md new file mode 100644 index 00000000..c179b1cf --- /dev/null +++ b/docs/architecture/management.md @@ -0,0 +1,122 @@ +# Fabric Management + +The installation of a Hedgehog Fabric deployment is carried out using Fabricator (hhfab CLI). Once deployed, ongoing operations are managed via the [Kubernetes](https://kubernetes.io) CLI, [`kubectl`](https://kubernetes.io/docs/reference/kubectl/). + +In this workflow, the Kubernetes API Server processes the Fabric Custom Resources (CRs) and forwards them to the Fabric Controller. The Fabric Controller then generates the required SONiC configurations and dispatches them to the Fabric Agent, which applies these configurations to the SONiC Config DB via gNMI. Simultaneously, Alloy collects metrics and logs from the SONiC switches, sending this information to the monitoring tools for continuous monitoring. + +The diagram below illustrates the general workflow for fabric management as well as the interactions between control plane components and the SONiC switches that form the Fabric. + +```mermaid +graph TD + +%% Define the nodes (General Workflow excludes Git & ArgoCD) +User[User] +Kubectl[kubectl] +Fabricator[Fabricator] + +%% Control Node components +subgraph ControlNode["Control Node"] + K8S[Kubernetes API Server] + FC[Fabric Controller] + K9s[K9s] + KubectlFabric["kubectl fabric"] +end + +%% SONiC Switch components +subgraph SONiCSwitch["SONiC Switch"] + FA[Fabric Agent] + Alloy[Alloy] + DB[SONiC Config DB] +end + +%% External monitoring +Monitoring[Loki/Grafana/Tempo/Mimir] + +%% Define the relationships +Kubectl -.->|Direct kubectl commands| K8S +User -.->|CLI| Fabricator +User -.->|CLI| Kubectl +Kubectl -.->|Plugin| KubectlFabric +Fabricator -->|SSH access| K9s +Fabricator -->|Applies CRs| K8S +KubectlFabric -->|Interacts with| K8S +K9s -->|UI Manages| K8S +K8S -->|Sends CRDs| FC +FC -->|Generates SONiC Configs| FA +FA -->|Applies Config via gNMI| DB +FA -->|Reports Metrics| Alloy +Alloy -->|Sends Logs & Metrics| Monitoring +``` + +--- + +## **Management Workflow Overview** + +### **User** +- **Creates Fabric CR YAMLs** and commits them to version control. +- **Directly interacts with SONiC switches** via the Fabricator CLI. +- **Uses [`kubectl`](https://kubernetes.io/docs/reference/kubectl/) and `kubectl fabric`** to interact with the Kubernetes API for fabric resource management. + +### **Kubernetes API Server (K8S)** +- Part of [Kubernetes](https://kubernetes.io). +- Manages Fabric Custom Resources (CRs) and interacts with the **Fabric Controller**. + +### **kubectl & kubectl fabric** +- [`kubectl`](https://kubernetes.io/docs/reference/kubectl/) is the standard CLI tool for [Kubernetes](https://kubernetes.io). +- `kubectl fabric` is a plugin that extends `kubectl` with fabric-specific commands and interacts with the Kubernetes API Server. + +### **Fabricator** +- CLI tool that provides direct interaction with the Kubernetes API. +- Can apply configurations via **SSH access** (using **K9s**) or by directly managing Fabric CRs with YAML files. + +### **K9s** +- A Kubernetes UI that provides visibility and control over Kubernetes resources, allowing you to manage pods, nodes, services, logs, and Fabric resources efficiently. It includes some helper plugins: + - **SSH** – SSH into a fabric switch + - **Serial** – Open a serial connection to a fabric switch + - **Reboot** – Reboot a fabric switch + - **Power Reset** Perform a power reset on a fabric switch in the NOS + - **Reinstall** – Reinstall a fabric switch + +### **SONiC Switch Relevant Components** +- **Fabric Agent:** Receives configurations from the Fabric Controller and applies them to the SONiC switches via gNMI. +- **Alloy:** Monitors SONiC and reports metrics. +- **SONiC Config DB:** Stores and manages switch configuration data. + +### **Monitoring** +- Logs and metrics from SONiC are collected and sent to [Loki](https://grafana.com/oss/loki/) and [Mimir](https://grafana.com/oss/mimir/) for visualization and analysis through [Grafana](https://grafana.com). + +--- + +## **GitOps Functionality (ArgoCD)** + +GitOps workflows can be leveraged using [ArgoCD](https://argo-cd.readthedocs.io/en/stable/). This is an alternative approach to show that a Fabric can be used with industry standard tools seamlessly. + +- **User Actions:** + - The user **creates Fabric CR YAMLs** and pushes them to a [Git repository](https://git-scm.com) for version control. +- **ArgoCD Actions:** + - [ArgoCD](https://argo-cd.readthedocs.io/en/stable/) monitors the Git repository. + - ArgoCD **pulls the CRs from Git** and applies them to [Kubernetes](https://kubernetes.io) via the Kubernetes API Server. + +```mermaid +graph TD + +%% Define the nodes (GitOps includes Git & ArgoCD) +User[User] +Kubectl[kubectl] + Git[Git Repository] +ArgoCD[ArgoCD] + +%% Control Node components +subgraph ControlNode["Control Node"] + K8S[Kubernetes API Server] + FC[Fabric Controller] +end + +%% Define the relationships +User -->|Fabric CR YAMLs| Git +Kubectl -.->|Direct kubectl commands| K8S +User -.->|CLI| Kubectl +Git -.->|ArgoCD pulls| ArgoCD +ArgoCD -->|Applies CRs| K8S +K8S -->|Sends CRDs| FC +``` From 3b76341be0ce78ebcb06ef0319973dc89479576b Mon Sep 17 00:00:00 2001 From: Pau Capdevila Date: Mon, 24 Mar 2025 11:26:01 +0100 Subject: [PATCH 2/6] chore: refine and deduplicate architecture pages Signed-off-by: Pau Capdevila --- docs/architecture/management.md | 46 ++++++++++----------------------- docs/architecture/overview.md | 2 ++ 2 files changed, 15 insertions(+), 33 deletions(-) diff --git a/docs/architecture/management.md b/docs/architecture/management.md index c179b1cf..c8079e67 100644 --- a/docs/architecture/management.md +++ b/docs/architecture/management.md @@ -1,10 +1,12 @@ # Fabric Management +This section focuses on operational aspects of Hedgehog Fabric, explaining available tools and typical workflows for administrators. It builds upon the architectural concepts explained in the [Overview](overview.md). + The installation of a Hedgehog Fabric deployment is carried out using Fabricator (hhfab CLI). Once deployed, ongoing operations are managed via the [Kubernetes](https://kubernetes.io) CLI, [`kubectl`](https://kubernetes.io/docs/reference/kubectl/). -In this workflow, the Kubernetes API Server processes the Fabric Custom Resources (CRs) and forwards them to the Fabric Controller. The Fabric Controller then generates the required SONiC configurations and dispatches them to the Fabric Agent, which applies these configurations to the SONiC Config DB via gNMI. Simultaneously, Alloy collects metrics and logs from the SONiC switches, sending this information to the monitoring tools for continuous monitoring. +In this workflow, the Kubernetes API Server processes the Fabric Custom Resources (CRs) and forwards them to the Fabric Controller. The Fabric Controller then translates these high-level network intents into concrete network configurations and manages their application to the underlying infrastructure. The Controller continuously monitors the state of the network resources, ensuring they match the desired state defined in the CRs. -The diagram below illustrates the general workflow for fabric management as well as the interactions between control plane components and the SONiC switches that form the Fabric. +The diagram below illustrates the general workflow for fabric management: ```mermaid graph TD @@ -22,16 +24,6 @@ subgraph ControlNode["Control Node"] KubectlFabric["kubectl fabric"] end -%% SONiC Switch components -subgraph SONiCSwitch["SONiC Switch"] - FA[Fabric Agent] - Alloy[Alloy] - DB[SONiC Config DB] -end - -%% External monitoring -Monitoring[Loki/Grafana/Tempo/Mimir] - %% Define the relationships Kubectl -.->|Direct kubectl commands| K8S User -.->|CLI| Fabricator @@ -42,10 +34,6 @@ Fabricator -->|Applies CRs| K8S KubectlFabric -->|Interacts with| K8S K9s -->|UI Manages| K8S K8S -->|Sends CRDs| FC -FC -->|Generates SONiC Configs| FA -FA -->|Applies Config via gNMI| DB -FA -->|Reports Metrics| Alloy -Alloy -->|Sends Logs & Metrics| Monitoring ``` --- @@ -53,7 +41,7 @@ Alloy -->|Sends Logs & Metrics| Monitoring ## **Management Workflow Overview** ### **User** -- **Creates Fabric CR YAMLs** and commits them to version control. +- **Creates Fabric CR YAMLs** and applies them through standard Kubernetes resource management. - **Directly interacts with SONiC switches** via the Fabricator CLI. - **Uses [`kubectl`](https://kubernetes.io/docs/reference/kubectl/) and `kubectl fabric`** to interact with the Kubernetes API for fabric resource management. @@ -71,19 +59,11 @@ Alloy -->|Sends Logs & Metrics| Monitoring ### **K9s** - A Kubernetes UI that provides visibility and control over Kubernetes resources, allowing you to manage pods, nodes, services, logs, and Fabric resources efficiently. It includes some helper plugins: - - **SSH** – SSH into a fabric switch - - **Serial** – Open a serial connection to a fabric switch - - **Reboot** – Reboot a fabric switch - - **Power Reset** Perform a power reset on a fabric switch in the NOS - - **Reinstall** – Reinstall a fabric switch - -### **SONiC Switch Relevant Components** -- **Fabric Agent:** Receives configurations from the Fabric Controller and applies them to the SONiC switches via gNMI. -- **Alloy:** Monitors SONiC and reports metrics. -- **SONiC Config DB:** Stores and manages switch configuration data. - -### **Monitoring** -- Logs and metrics from SONiC are collected and sent to [Loki](https://grafana.com/oss/loki/) and [Mimir](https://grafana.com/oss/mimir/) for visualization and analysis through [Grafana](https://grafana.com). + - **SSH** – SSH into a fabric switch + - **Serial** – Open a serial connection to a fabric switch + - **Reboot** – Reboot a fabric switch + - **Power Reset** Perform a power reset on a fabric switch in the NOS + - **Reinstall** – Reinstall a fabric switch --- @@ -92,10 +72,10 @@ Alloy -->|Sends Logs & Metrics| Monitoring GitOps workflows can be leveraged using [ArgoCD](https://argo-cd.readthedocs.io/en/stable/). This is an alternative approach to show that a Fabric can be used with industry standard tools seamlessly. - **User Actions:** - - The user **creates Fabric CR YAMLs** and pushes them to a [Git repository](https://git-scm.com) for version control. + - The user **creates Fabric CR YAMLs** and pushes them to a [Git repository](https://git-scm.com) for version control. - **ArgoCD Actions:** - - [ArgoCD](https://argo-cd.readthedocs.io/en/stable/) monitors the Git repository. - - ArgoCD **pulls the CRs from Git** and applies them to [Kubernetes](https://kubernetes.io) via the Kubernetes API Server. + - [ArgoCD](https://argo-cd.readthedocs.io/en/stable/) monitors the Git repository. + - ArgoCD **pulls the CRs from Git** and applies them to [Kubernetes](https://kubernetes.io) via the Kubernetes API Server. ```mermaid graph TD diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 4cc24d79..79e1a6e4 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -1,5 +1,7 @@ # Overview +This section provides an architectural understanding of Hedgehog Open Network Fabric, explaining its components and their relationships. + Hedgehog Open Network Fabric leverages the Kubernetes API to manage its resources. All user-facing APIs are exposed as Kubernetes Custom Resources Definitions (CRDs), allowing users to manage Fabric resources using standard Kubernetes tools. To make network switches Kubernetes-aware, the Fabric employs an **Agent** running on each switch. This agent acts as an interface between the Kubernetes control plane and the switch internal network configuration mechanisms. It continuously syncs desired state from Kubernetes via the Fabric Controller and applies configurations using **gNMI** (gRPC Network Management Interface). From 88b640e769d27833b76c42156b088fc34b9b05fe Mon Sep 17 00:00:00 2001 From: Pau Capdevila Date: Tue, 15 Apr 2025 18:07:28 +0200 Subject: [PATCH 3/6] chore: expand kubectl fabric commands Signed-off-by: Pau Capdevila --- docs/architecture/management.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/architecture/management.md b/docs/architecture/management.md index c8079e67..cc7d7ffd 100644 --- a/docs/architecture/management.md +++ b/docs/architecture/management.md @@ -51,7 +51,21 @@ K8S -->|Sends CRDs| FC ### **kubectl & kubectl fabric** - [`kubectl`](https://kubernetes.io/docs/reference/kubectl/) is the standard CLI tool for [Kubernetes](https://kubernetes.io). -- `kubectl fabric` is a plugin that extends `kubectl` with fabric-specific commands and interacts with the Kubernetes API Server. +- `kubectl fabric` is a plugin that extends `kubectl` with fabric-specific commands and interacts with the Kubernetes API Server. It supports the following commands: + - **vpc** – Manage VPCs: create, attach, peer, or wipe all related resources + - **switch** – Manage switches: IP, SSH, serial, reboot, reinstall + - **connection** – View connections: management, fabric, vpc-loopback + - **switchgroup** – Create SwitchGroups + - **external** – Manage external BGP peers and VPC peering + - **wiring** – Export fabric wiring diagrams + - **inspect** – Inspect and troubleshoot fabric resources: + - `fabric` – Overview of control nodes and switches + - `switch`, `port` – Status, counters, and usage + - `server`, `connection` – VPC/external attachments and link details + - `vpc` – Subnet attachments and reachability + - `bgp`, `lldp` – Neighbor status checks + - `ip`, `mac` – Lookup usage across the fabric + - `access` – Check reachability between VPCs, servers, and IPs ### **Fabricator** - CLI tool that provides direct interaction with the Kubernetes API. From 6c1492b580c43824ef762fe626acf8587a2f5147 Mon Sep 17 00:00:00 2001 From: Pau Capdevila Date: Wed, 16 Apr 2025 10:17:51 +0200 Subject: [PATCH 4/6] chore: clarify workflows and diagrams Signed-off-by: Pau Capdevila --- docs/architecture/management.md | 58 +++++++++++++++++++-------------- 1 file changed, 33 insertions(+), 25 deletions(-) diff --git a/docs/architecture/management.md b/docs/architecture/management.md index cc7d7ffd..3225ffcc 100644 --- a/docs/architecture/management.md +++ b/docs/architecture/management.md @@ -6,15 +6,18 @@ The installation of a Hedgehog Fabric deployment is carried out using Fabricator In this workflow, the Kubernetes API Server processes the Fabric Custom Resources (CRs) and forwards them to the Fabric Controller. The Fabric Controller then translates these high-level network intents into concrete network configurations and manages their application to the underlying infrastructure. The Controller continuously monitors the state of the network resources, ensuring they match the desired state defined in the CRs. +In this workflow, the user interacts with the Kubernetes API Server by creating or modifying Fabric Custom Resources (CRs). The Fabric Controller watches the Kubernetes API for changes to these CRs. Upon detecting a change, the Controller translates the high-level network intent into concrete configurations and applies them to the fabric switches. + +The Controller continuously reconciles the actual network state with the desired state declared in the CRs, ensuring consistency and reliability across the fabric. + The diagram below illustrates the general workflow for fabric management: ```mermaid graph TD -%% Define the nodes (General Workflow excludes Git & ArgoCD) +%% Define the nodes User[User] Kubectl[kubectl] -Fabricator[Fabricator] %% Control Node components subgraph ControlNode["Control Node"] @@ -26,14 +29,12 @@ end %% Define the relationships Kubectl -.->|Direct kubectl commands| K8S -User -.->|CLI| Fabricator +User -.->|SSH Control| K9s User -.->|CLI| Kubectl -Kubectl -.->|Plugin| KubectlFabric -Fabricator -->|SSH access| K9s -Fabricator -->|Applies CRs| K8S +User -.->|SSH Control| KubectlFabric KubectlFabric -->|Interacts with| K8S K9s -->|UI Manages| K8S -K8S -->|Sends CRDs| FC +FC -->|Watches CRDs| K8S ``` --- @@ -42,7 +43,6 @@ K8S -->|Sends CRDs| FC ### **User** - **Creates Fabric CR YAMLs** and applies them through standard Kubernetes resource management. -- **Directly interacts with SONiC switches** via the Fabricator CLI. - **Uses [`kubectl`](https://kubernetes.io/docs/reference/kubectl/) and `kubectl fabric`** to interact with the Kubernetes API for fabric resource management. ### **Kubernetes API Server (K8S)** @@ -53,7 +53,7 @@ K8S -->|Sends CRDs| FC - [`kubectl`](https://kubernetes.io/docs/reference/kubectl/) is the standard CLI tool for [Kubernetes](https://kubernetes.io). - `kubectl fabric` is a plugin that extends `kubectl` with fabric-specific commands and interacts with the Kubernetes API Server. It supports the following commands: - **vpc** – Manage VPCs: create, attach, peer, or wipe all related resources - - **switch** – Manage switches: IP, SSH, serial, reboot, reinstall + - **switch** – Manage switches: get IP, SSH, serial, reboot, reinstall - **connection** – View connections: management, fabric, vpc-loopback - **switchgroup** – Create SwitchGroups - **external** – Manage external BGP peers and VPC peering @@ -81,36 +81,44 @@ K8S -->|Sends CRDs| FC --- -## **GitOps Functionality (ArgoCD)** +## **GitOps Integration** + +GitOps is a continuous deployment approach where the desired state of a Kubernetes cluster is stored in Git. A GitOps controller monitors this repository and ensures the actual cluster state matches the declared state, using the Kubernetes API. + +Hedgehog Fabric is fully compatible with GitOps workflows and can be managed declaratively using any GitOps-compatible tool such as [ArgoCD](https://argo-cd.readthedocs.io/en/stable/) or [Flux](https://fluxcd.io). -GitOps workflows can be leveraged using [ArgoCD](https://argo-cd.readthedocs.io/en/stable/). This is an alternative approach to show that a Fabric can be used with industry standard tools seamlessly. +- **User Responsibilities:** + - Define Fabric Custom Resources (CRs) in YAML. + - Commit and push these CRs to a [Git repository](https://git-scm.com) for version control and collaboration. -- **User Actions:** - - The user **creates Fabric CR YAMLs** and pushes them to a [Git repository](https://git-scm.com) for version control. -- **ArgoCD Actions:** - - [ArgoCD](https://argo-cd.readthedocs.io/en/stable/) monitors the Git repository. - - ArgoCD **pulls the CRs from Git** and applies them to [Kubernetes](https://kubernetes.io) via the Kubernetes API Server. +- **GitOps Tool Responsibilities:** + - Monitor the Git repository for updates. + - Apply changes to the cluster automatically via the Kubernetes API Server. + +This enables seamless integration of Hedgehog Fabric into modern CI/CD pipelines and Git-centric workflows. ```mermaid graph TD -%% Define the nodes (GitOps includes Git & ArgoCD) -User[User] +%% Define the nodes +User[User] Kubectl[kubectl] - Git[Git Repository] -ArgoCD[ArgoCD] - +Git[Git Repository] +GitOps[GitOps] %% Control Node components subgraph ControlNode["Control Node"] K8S[Kubernetes API Server] FC[Fabric Controller] end - %% Define the relationships User -->|Fabric CR YAMLs| Git Kubectl -.->|Direct kubectl commands| K8S User -.->|CLI| Kubectl -Git -.->|ArgoCD pulls| ArgoCD -ArgoCD -->|Applies CRs| K8S -K8S -->|Sends CRDs| FC +Git -.->|GitOps pulls| GitOps +GitOps -->|Applies CRs| K8S +%% Keep rel with empty text to keep layout +K8S -->| | FC +FC -->|Watches CRDs| K8S +%% Style the original arrow invisible +linkStyle 5 stroke:none,fill:none; ``` From ca4a1fcedda6676b6834c59edc7ccddcca856ec6 Mon Sep 17 00:00:00 2001 From: Pau Capdevila Date: Wed, 16 Apr 2025 10:29:03 +0200 Subject: [PATCH 5/6] chore: remove incorrect fabricator workflows Signed-off-by: Pau Capdevila --- docs/architecture/management.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/architecture/management.md b/docs/architecture/management.md index 3225ffcc..608aeabe 100644 --- a/docs/architecture/management.md +++ b/docs/architecture/management.md @@ -68,11 +68,10 @@ FC -->|Watches CRDs| K8S - `access` – Check reachability between VPCs, servers, and IPs ### **Fabricator** -- CLI tool that provides direct interaction with the Kubernetes API. -- Can apply configurations via **SSH access** (using **K9s**) or by directly managing Fabric CRs with YAML files. +- Using the **hhfab** CLI you can **SSH** the control node and use **K9s** or **kubectl** and **kubectl fabric** to manage or monitor Fabric API objects. ### **K9s** -- A Kubernetes UI that provides visibility and control over Kubernetes resources, allowing you to manage pods, nodes, services, logs, and Fabric resources efficiently. It includes some helper plugins: +- K9s is a terminal based Kubernetes UI that provides visibility and control over Kubernetes resources, allowing you to manage pods, nodes, services, logs, and Fabric resources efficiently. It includes some helper plugins: - **SSH** – SSH into a fabric switch - **Serial** – Open a serial connection to a fabric switch - **Reboot** – Reboot a fabric switch From 857ac1d2fc80a9870ec453eaaf39532e5f29ed6a Mon Sep 17 00:00:00 2001 From: Pau Capdevila Date: Wed, 16 Apr 2025 10:49:51 +0200 Subject: [PATCH 6/6] chore: add clarification about kubectl Signed-off-by: Pau Capdevila --- docs/architecture/management.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/architecture/management.md b/docs/architecture/management.md index 608aeabe..ea898e8c 100644 --- a/docs/architecture/management.md +++ b/docs/architecture/management.md @@ -24,17 +24,20 @@ subgraph ControlNode["Control Node"] K8S[Kubernetes API Server] FC[Fabric Controller] K9s[K9s] + KubectlK3s[kubectl] KubectlFabric["kubectl fabric"] end %% Define the relationships -Kubectl -.->|Direct kubectl commands| K8S +Kubectl -.->|Sends CRs and queries| K8S +KubectlK3s -.->|Sends CRs and queries| K8S +KubectlK3s -.->|Invokes| KubectlFabric +User -.->|SSH Control| KubectlK3s User -.->|SSH Control| K9s User -.->|CLI| Kubectl -User -.->|SSH Control| KubectlFabric -KubectlFabric -->|Interacts with| K8S -K9s -->|UI Manages| K8S -FC -->|Watches CRDs| K8S +KubectlFabric -->|Applies/inspects CRs| K8S +K9s -->|Reads/writes CRs via API| K8S +FC -->|Watches CRs| K8S ``` --- @@ -44,6 +47,7 @@ FC -->|Watches CRDs| K8S ### **User** - **Creates Fabric CR YAMLs** and applies them through standard Kubernetes resource management. - **Uses [`kubectl`](https://kubernetes.io/docs/reference/kubectl/) and `kubectl fabric`** to interact with the Kubernetes API for fabric resource management. + - Can use their **own `kubectl` installation** or the one pre-installed on the control node as part of [k3s](https://k3s.io). ### **Kubernetes API Server (K8S)** - Part of [Kubernetes](https://kubernetes.io). @@ -117,7 +121,7 @@ Git -.->|GitOps pulls| GitOps GitOps -->|Applies CRs| K8S %% Keep rel with empty text to keep layout K8S -->| | FC -FC -->|Watches CRDs| K8S +FC -->|Watches CRs| K8S %% Style the original arrow invisible linkStyle 5 stroke:none,fill:none; ```