k8s version for rodan

[homework36]

Part 1

We first wrap all these steps (https://github.com/DDMAL/Rodan/wiki/Working-on-Rodan) with k8s.
The yml file is here: https://github.com/DDMAL/Rodan/blob/k8s-local-test/kubernetes-local.yml

Steps to deploy locally on Mac mini
Prerequisites (one-time)

# Install gettext for envsubst
brew install gettext
brew link --force gettext

# Enable Kubernetes in Docker Desktop:
# Docker Desktop → Settings → Kubernetes → Enable Kubernetes → Apply & Restart

# Confirm context is pointing at Docker Desktop
kubectl config use-context docker-desktop

Deploy

cd /path/to/local/Rodan

export RODAN_SRC=/path/to/local/Rodan
envsubst < kubernetes-local.yml | kubectl apply -f -

Wait for pods

kubectl get pods -n rodan-local -w

Wait until all 10 pods show Running. Takes about 1-2 minutes.

Start services (each in its own terminal tab)

Tab 1 — Rodan API:
kubectl exec -it -n rodan-local deploy/rodan-main -- /run/start

Tab 2 — Celery (core tasks):
kubectl exec -it -n rodan-local deploy/celery -- /run/start-celery

Tab 3 — Python3 jobs:
kubectl exec -it -n rodan-local deploy/py3-celery -- /run/start-celery

Access at http://localhost:/api

Stop

kubectl delete namespace rodan-local

Current it still looks like this

Mac mini
  └── Docker Desktop (runs the Kubernetes node)
      └── Kubernetes cluster
          ├── pod: nginx          → container from ddmal/nginx
          ├── pod: rodan-main     → container from ddmal/rodan-main
          ├── pod: celery         → container from ddmal/rodan-main
          ├── pod: py3-celery     → container from ddmal/rodan-python3-celery
          ├── pod: postgres       → container from ddmal/postgres-plpython
          └── ...

[homework36]

Part 2

Currently, the local Kubernetes setup (kubernetes-local.yml) pulls pre-built Docker images for each service. While this works, Docker images tend to be large and require Docker Desktop (or a Docker daemon) to build and run, adding overhead for local development. We want to explore whether there are lighter-weight alternatives, such as building minimal OCI-compliant images with tools like Podman or Buildah (daemonless, no Docker daemon required), or using distroless/slim base images to reduce image size. The goal is to keep the Kubernetes-based local dev workflow but reduce the dependency on Docker Desktop and the storage/bandwidth cost of large images. This is worth investigating before committing further to the current image-based approach.

Replace Docker Desktop → Podman
Podman is daemonless, rootless, and free. It's OCI-compatible so kubernetes-local.yml works as-is — no manifest changes needed. Podman Desktop has built-in Kubernetes support. This removes the Docker Desktop dependency entirely without changing the workflow.

Steps to deploy on local Mac mini
(All yaml files are on the k8s-local-test branch.)
1.Install Podman Desktop

brew install podman podman-desktop

2. Initialize and start the Podman machine (the lightweight VM that replaces Docker Desktop)

podman machine init --cpus 4 --memory 6144
  podman machine start

3. Open Podman Desktop and enable Kubernetes

  Podman Desktop → Settings → Kubernetes → Create a Kind cluster

This sets up a lightweight Kubernetes cluster using Kind backed by Podman instead of Docker.

4. Switch kubectl context

kubectl config use-context kind-kind

5. Deploy

cd /path/to/local/Rodan

export RODAN_SRC=/path/to/local/Rodan
envsubst < kind-config.yml | kind create cluster --name rodan --config -

# Switch context
kubectl config use-context kind-rodan

# Deploy
kubectl apply -f kubernetes-podman.yml

# Wait for pods
kubectl get pods -n rodan-local -w

6. Once all pods are Running, in separate tabs:

kubectl exec -it -n rodan-local deploy/rodan-main -- /run/start
kubectl exec -it -n rodan-local deploy/celery -- /run/start-celery
kubectl exec -it -n rodan-local deploy/py3-celery -- /run/start-celery

7. App available at http://localhost:30080/.

Switching from Docker Desktop to Podman + Kind removes the dependency on the Docker daemon and Docker Desktop's licensing requirements, and provides a daemonless, rootless container runtime that is lighter on background resource usage.

However, the core limitation remains unchanged: the setup still pulls the same large pre-built OCI images from Docker Hub (ddmal/rodan-main 17.8GB, ddmal/rodan-python3-celery 13.7GB), so disk usage and pull times are identical to the Docker Desktop approach. Podman also adds setup friction compared to Docker Desktop, since Kind clusters require explicit port mappings and host directory mounts configured at creation time, and NodePort services don't bind to localhost automatically. The meaningful next step toward a truly lightweight setup is rebuilding the ddmal/* images with slimmer base images and multi-stage builds, which is independent of the runtime choice.

[homework36]

Clarification from GPT:

Building images vs running images

• Docker/Podman — tools you use on your laptop to build images and run them locally
• containerd/CRI-O — lightweight runtimes that only run images, used inside production K8s nodes

When you deploy to EKS/GKE, the cloud nodes never use Docker or Podman. They use containerd directly, which is the low-level piece that actually starts and stops containers. Docker itself is just a user-friendly wrapper around containerd.

Why Kubernetes dropped Docker in 1.24

Kubernetes talks to container runtimes through a standard interface called CRI (Container Runtime Interface). Docker didn't implement CRI natively, so Kubernetes had to ship a translation layer called dockershim to bridge them.
That was dead weight — extra complexity, extra attack surface, extra maintenance. They removed it and told everyone to use containerd or CRI-O directly, which both natively speak CRI.

The mental model

  Your laptop (dev/build)          Production K8s node
  ─────────────────────            ──────────────────────
  Docker or Podman                 containerd or CRI-O
      ↓ builds                         ↓ runs
  OCI image ──── push to registry ──── pull & run

The image format (OCI) is the common language. You build on your laptop with whatever tool you like, push to a registry, and production pulls and runs it with containerd — completely unaware of how it was built.

So Docker/Podman are essentially dev tools in this picture. The production runtime is always something leaner.

Follow-up question

Does it mean if we use the k8s setup, on the production instance we don't need docker or podman (unlike deployment using docker swarm)?

Answer from GPT

Yes, exactly. That's one of the key advantages of committing to Kubernetes.

Docker Swarm / docker-compose in production:
Every production server needs:

• Docker daemon running
• docker-compose or swarm configured
• Docker Desktop or Docker Engine installed

Kubernetes in production (EKS/GKE/your own cluster):
Production nodes only need:

• containerd (usually pre-installed)
• kubelet
  (No Docker, no Podman, nothing else)

You build the image once on a dev machine or CI pipeline, push it to a registry (Docker Hub, ECR, GCR), and the K8s nodes just pull and run it via containerd. The nodes themselves have no knowledge of Docker.

My conclusion

We are exploring a new groundwork for eventually migrating production off Swarm entirely and it should be a meaningful long-term improvement for the project.

Advantages (summarized by GPT)

Operations

• No Docker daemon to manage on production servers — fewer moving parts, less to break
• Built-in rolling updates, health checks, and auto-restart without custom scripting
• Fine-grained resource limits per service (currently rodan-main and celery workers could starve each other on Swarm)

Scaling

• Celery workers (py3-celery, gpu-celery) could scale up/down independently based on job queue depth — Swarm can scale too but K8s has better tooling for this (HorizontalPodAutoscaler)
• GPU jobs could be scheduled only on nodes that actually have GPUs via node selectors — much harder to express in Swarm

Developer experience

• Local dev and production use the same paradigm — what works in kubernetes-podman.yml locally behaves the same way in a production cluster
• Currently there's a big gap: local uses docker-compose, production uses Swarm, they diverge over time

Ecosystem

• Swarm is effectively in maintenance mode — Docker hasn't invested in it for years
• Kubernetes has a massive ecosystem: Helm charts, monitoring (Prometheus/Grafana), logging (Loki), secret management (Vault) all integrate natively
• Most cloud infrastructure tooling (Terraform, ArgoCD, Flux) targets Kubernetes, not Swarm

@amirsadraabdollahi We would really appreciate your thoughts and expertise on this! Even though we are going to develop a new system/server, many features/workflows in this old repo might still be relevant.