Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Small Update to "Writing a Kubernetes Operator" #104

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Small Update to "Writing a Kubernetes Operator" #104

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ff28503
..
DmitryDodzin Oct 22, 2024
b300918
Apply suggestions from code review
DmitryDodzin Nov 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions content/en/blog/kubernetes-operator/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ tags:
- operator
- cloud
date: 2023-03-09T0:00:00+00:00
lastmod: 2023-03-09T0:00:00+00:00
lastmod: 2024-10-22T0:00:00+00:00
draft: false
weight: 50
contributors: ["Dmitry Dodzin"]
Expand All @@ -17,20 +17,21 @@ As part of mirrord For Teams, we wanted to build a persistent component that wou
1. **Kubernetes-native** - meaning it leverages the Kubernetes APIs and ecosystem
2. **Cluster-synchronized** - Manage and synchronize the use of our open-source project, mirrord, from the cluster’s point of view.

Some research pointed us in the direction of the Kubernetes Operator/Controller pattern.
The operator pattern can be quite ambiguous, and we found the guides that currently exist for it to be quite dense and technical. In this post, I want to instead take a step-by-step approach and provide a quick start for newcomers looking to explore the operator pattern.
Some research pointed us in the direction of the Kubernetes Operator pattern.
The operator pattern can be quite ambiguous, and we found the guides that currently exist for it to be quite dense and technical. In this post, I want to instead take a step-by-step approach and provide a quick start for newcomers looking to explore the APIService flavor of the operator pattern.

## Why would you need to write an operator/controller?
On many occasions, the Deployment or StatefulSet at the core of your product will not be self-sufficient but need to access other resources in the cluster. For example, it might need to share a persistent volume across deployments, read a certificate from a Secret, or rely on a headless service for discovery logic. These can be achieved through manual configurations or using a Helm chart or Kustomize template, but then your component is poorly abstracted, so prone to misconfiguration by your users and harder to update.

Using a Kubernetes operator/controller can make it easier for your users to setup and configure your product on their cluster. Let’s illustrate this with an example: [CockroachDB](https://www.cockroachlabs.com/) is a sharded database with a Postgres-compatible API. Unlike PostgreSQL, it has some safety features enabled by default like requiring SSL encrypted connections for writes, so to deploy CockroachDB you would theoretically need to create and maintain a certificate for each of its Deployments on your Kubernetes cluster. For this reason, they created cockroach-operator. Once installed, a new resource named CrdbCluster becomes available. Whenever the user wants to create a new CockroachDB cluster, they now only have to create a new CrdbCluster object, and the [cockroach-operator](https://github.com/cockroachdb/cockroach-operator) takes care of the rest.
Using Kubernetes operators and CRDs can make it easier for your users to setup and configure your product on their cluster. Let’s illustrate this with an example: [CockroachDB](https://www.cockroachlabs.com/) is a sharded database with a Postgres-compatible API. Unlike PostgreSQL, it has some safety features enabled by default like requiring SSL encrypted connections for writes, so to deploy CockroachDB you would theoretically need to create and maintain a certificate for each of its Deployments on your Kubernetes cluster. For this reason, they created cockroach-operator. Once installed, a new resource named CrdbCluster becomes available. Whenever the user wants to create a new CockroachDB cluster, they now only have to create a new CrdbCluster object, and the [cockroach-operator](https://github.com/cockroachdb/cockroach-operator) takes care of the rest.

## Operator vs. Controller
## APIService vs. Controller

A **controller** is a software component that tracks Kubernetes objects and interacts with them. The objects themselves are managed by Kubernetes itself. For example, Admission Controllers watch new objects being created and enforce policies on them. The objects the controller manages can be existing objects. Note that the controller is a pattern. It doesn’t dictate how the controller should run - it can be from a desktop, server, cluster, or anywhere else where it can interact with the Kubernetes API.

An **operator** is a controller that tracks new resources you can add by using CustomResourceDefinition.
An operator can use the Kubernetes API to manage these resources; alternatively, a third component called APIService can be leveraged for handling requests to these resources to the Kubernetes API.
An **APIService** is an extension to actual kubelet-api that allows you to create virtual CustomResourceDefinitions that don't exist in etcd and allows you to implement more advanced verbs manualy or use the kubernetes api to expose more advanced api's like `kubectl auth can-i` api that uses `SelfSubjectAccessReview` (it's a `create` only api that expects no or an empty `metadata` field).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
An **APIService** is an extension to actual kubelet-api that allows you to create virtual CustomResourceDefinitions that don't exist in etcd and allows you to implement more advanced verbs manualy or use the kubernetes api to expose more advanced api's like `kubectl auth can-i` api that uses `SelfSubjectAccessReview` (it's a `create` only api that expects no or an empty `metadata` field).
An **APIService** is an extension to actual Kubelet API that allows you to:
- Create virtual CustomResourceDefinitions that don't exist in etcd
- Implement more advanced verbs manually
- Use the Kubernetes API to expose more APIs like `kubectl auth can-i` api that uses `SelfSubjectAccessReview` (a `create` only api that expects no or an empty `metadata` field).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DmitryDodzin wdym by "expects no"?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

like create the request doesn't even have the metadata field like only the apiVersion, kind and spec this usually will result in an error because metadata is mandatory for most resources


For "mirrord for Teams" we use a mixture of the APIService and Controller patterns where some of our CustomResourceDefinitions are managed by Controllers and some are exposed via an APIService.

## Possible languages and frameworks

Expand All @@ -57,7 +58,7 @@ For the example in this post, we will use Rust + kube-rs. Here are a few reasons

## This is where the tutorial starts

In the sections that follow, we’ll be creating an operator with an APIService. We’ll use Rust, but implementations in other languages can be extrapolated from it fairly easily. First, clone our [example repository](https://github.com/metalbear-co/farm-operator):
In the sections that follow, we’ll be creating an operator with an APIService. The Controller pattern is pretty well documented and is the more popular approach for writing operator, but if you do need the extra flexibility or unique features that APIService provides, this tutorial will explain what Kubernetes expects of your operator to be a valid APIService. We’ll use Rust, but implementations in other languages can be extrapolated from it fairly easily, as it's mainly an HTTPS server that is very similar to the actual Kubernetes API. First, clone our [example repository](https://github.com/metalbear-co/farm-operator):

```bash
git clone https://github.com/metalbear-co/farm-operator.git
Expand Down