Refactor Docker Compose to Kubernetes

[RonTuretzky]

Summary

This issue tracks the refactoring of the current Docker Compose setup to Kubernetes for improved production-readiness, scalability, and operational workflows.

Background

The current deployment infrastructure relies on Docker Compose for:

• Local development
• CI/CD E2E testing (integration-test.yml, local-integration-test.yml)
• 6 services: ethereum, eigenlayer, signer, node-1, node-2, node-3, router

Goals

1. Production-ready infrastructure: K8s manifests suitable for staging/production
2. Preserved E2E test coverage: All existing CI tests must continue to pass
3. Local development support: Easy local K8s setup using kind/minikube
4. Simplified operations: Leverage K8s primitives for health checks, restarts, scaling

Proposed Architecture

Kubernetes Resources

• Namespace: commonware-avs
• ConfigMap: avs-config (operators config, orchestrator configs)
• PersistentVolumeClaim: nodes-data (shared storage for keys and deployment artifacts)
• Job: eigenlayer-setup (init job for contract deployment and key generation)
• Deployment: ethereum, signer, router
• StatefulSet: avs-nodes (3 replicas with stable network identities)
• Services: ClusterIP for internal communication, NodePort for CI testing

Directory Structure

k8s/
├── base/
│   ├── kustomization.yaml
│   ├── namespace.yaml
│   ├── configmap.yaml
│   ├── pvc.yaml
│   ├── ethereum/
│   ├── eigenlayer/
│   ├── signer/
│   ├── nodes/
│   └── router/
└── overlays/
    ├── ci/
    └── local/

CI/CD Updates

• New workflow: k8s-integration-test.yml
• Uses helm/kind-action for ephemeral K8s clusters
• Preserves all existing test scenarios:
  • Counter increment test
  • Fast aggregation test (dev branch)
  • Ingress endpoint test (dev branch)

Test Coverage Preservation

Test	Current (Docker Compose)	New (Kubernetes)
Counter increment	✅ integration-test.yml	✅ k8s-integration-test.yml
Fast aggregation	✅ dev branch only	✅ dev branch only
Ingress endpoint	✅ dev branch only	✅ dev branch only
Local integration	✅ local-integration-test.yml	✅ kind + host router option

Implementation Tasks

• Create base Kubernetes manifests
• Create Kustomize overlays for CI and local
• Implement kind configuration
• Create k8s-integration-test.yml workflow
• Test all E2E scenarios
• Update documentation
• Optional: Keep docker-compose.yml for simple local dev

Full Spec

See docs/k8s-refactor-spec.md for the complete technical specification including:

• Detailed architecture diagrams
• Resource definitions
• CI/CD workflow details
• Edge cases and concessions
• Migration path

Open Questions

1. Production storage backend (EBS, GCE PD, local-path)?
2. External secrets integration (Vault, AWS Secrets Manager)?
3. Prometheus ServiceMonitor resources?
4. Network policies for pod-to-pod communication?
5. Resource limits for each service?

cc @dijarllozana