Skip to content

Commit 39b9b02

Browse files
AnastaZIukAnastaZIuk
committed
docs: streamline docs and add CI
1 parent f73e8f5 commit 39b9b02

7 files changed

Lines changed: 303 additions & 17 deletions

File tree

‎.github/workflows/terraform-quality-gate.yml‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
name: terraform-quality-gate
2+
3+
on:
4+
pull_request:
5+
branches: ["env/prod", "env/test"]
6+
push:
7+
branches: ["env/prod", "env/test"]
8+
9+
concurrency:
10+
group: terraform-quality-gate-${{ github.event.pull_request.base.ref || github.ref_name }}
11+
cancel-in-progress: false
12+
13+
permissions:
14+
contents: read
15+
16+
jobs:
17+
qa:
18+
runs-on: ubuntu-latest
19+
env:
20+
TF_IN_AUTOMATION: "true"
21+
defaults:
22+
run:
23+
shell: bash
24+
working-directory: terraform
25+
steps:
26+
- uses: actions/checkout@v4
27+
with:
28+
submodules: false
29+
fetch-depth: 1
30+
31+
- uses: hashicorp/setup-terraform@v3
32+
with:
33+
terraform_version: 1.6.6
34+
35+
- name: Terraform fmt
36+
run: terraform fmt -check -recursive
37+
38+
- name: Terraform init (no backend)
39+
run: terraform init -backend=false -input=false
40+
41+
- name: Terraform validate
42+
run: terraform validate -no-color
43+
44+
generate:
45+
needs: qa
46+
if: github.event_name == 'push' && github.actor != 'github-actions[bot]'
47+
runs-on: ubuntu-latest
48+
permissions:
49+
contents: write
50+
steps:
51+
- uses: actions/checkout@v4
52+
with:
53+
submodules: false
54+
fetch-depth: 0
55+
56+
- uses: terraform-docs/gh-actions@v1.3.0
57+
with:
58+
working-dir: terraform
59+
output-file: ../TERRAFORM.md
60+
output-method: replace
61+
62+
- name: Generate Kube diagram
63+
run: |
64+
docker run --rm \
65+
-v "$GITHUB_WORKSPACE":/work \
66+
philippemerle/kubediagrams \
67+
kube-diagrams -o KUBEDIAGRAM.png terraform/k8s/apps/kustomization.yaml
68+
69+
- name: Commit and push if changed
70+
run: |
71+
set -euo pipefail
72+
if [ -z "$(git status --porcelain -- TERRAFORM.md KUBEDIAGRAM.png)" ]; then
73+
echo "No changes to generated files."
74+
exit 0
75+
fi
76+
77+
git config user.name "github-actions[bot]"
78+
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
79+
git add TERRAFORM.md KUBEDIAGRAM.png
80+
git commit -m "chore(ci): update generated files [skip ci]"
81+
git push

‎README.md‎

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# TerraInfra
22

3-
Infra for Devsh (k3s on Scaleway) with GitOps via Flux. Branches:
3+
Infra for Devsh (k3s on Scaleway) with GitOps via Flux.
4+
5+
Branches:
46
- `env/prod` → production cluster (kept up, do not destroy)
57
- `env/test` → ephemeral/test cluster (can be recreated on demand)
68

@@ -17,10 +19,13 @@ Docs live in `docs/`:
1719
- `docs/security.md` – hardening matrix, how to add services with current security baseline
1820
- `docs/ui.md` – Kubernetes Dashboard (read-only) access
1921
- `docs/monitoring.md` – Grafana dashboard provisioning and updates
22+
- `docs/alerts.md` – alerting flow (Alertmanager → OnCall → Discord) + smoke tests
23+
- `docs/resources.md` – resource requests/limits, priority classes, uptime notes
24+
- `docs/ci.md` – CI checks and generated files
25+
26+
Generated by CI (do not edit manually):
27+
- `TERRAFORM.md` – Terraform inputs/outputs/module docs
28+
- `KUBEDIAGRAM.png` – Kubernetes architecture diagram from manifests
2029

21-
## Testing cheatsheet
22-
- Alerts → Discord: exec into alertmanager and send a test alert (labels `severity=warning|critical`) to verify Discord message + RESOLVED update:
23-
`k3s kubectl exec -n monitoring deploy/monitoring-kube-prometheus-alertmanager -- sh -c 'apk add --no-cache curl >/dev/null && curl -XPOST -H "Content-Type: application/json" -d '[{"\""}labels{"\""}:{"\""}alertname{"\""}:{"\""}TestAlert{"\""},"\""}severity{"\""}:{"\""}warning{"\""}},{"\""}annotations{"\""}:{"\""}summary{"\""}:{"\""}test{"\""}}]' http://localhost:9093/api/v2/alerts'`
24-
- Image rollout (website/blog): push a new semver tag to GHCR (`ghcr.io/devsh-graphics-programming/www-website` / `www-blog`), then watch automation:
25-
`flux get image policy -n flux-system`, `flux get image update -n flux-system`, confirm commit and rollout of website/blog Deployments.
26-
- Alert noise/dedupe: Alertmanager groups by alertname/namespace/cluster/resource with `for` on rules; routes only by `severity`.
30+
## Diagram
31+
![Kubernetes diagram](./KUBEDIAGRAM.png)

‎alerts.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Alerts
2+
3+
Moved to `docs/alerts.md`.

‎docs/alerts.md‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
# Alerting flow (OnCall)
2+
3+
- Routing: Prometheus Alertmanager → Grafana OnCall (two integrations: `critical`/`warning`) → OnCall outgoing webhook → `oncall-discord-proxy` → Discord webhooks (`#alerts-critical`, `#alerts-warning`).
4+
- Discord: one message per alert group, updated in-place on resolve; every message includes the `@OnCall` role mention.
5+
- OnCall stack lives in `monitoring-oncall` (HelmRelease `oncall`) with a dedicated Grafana UI: `https://${ENV_PREFIX}oncall.${BASE_DOMAIN}/grafana`.
6+
- Storage: Postgres + Redis subcharts (RabbitMQ is disabled); Grafana state on a local PV (`/mnt/data/oncall-grafana`).
7+
8+
## Bootstrap and wiring
9+
- Job `oncall-bootstrap` (namespace `monitoring-oncall`) runs on reconcile:
10+
- reads `alertmanager-oncall` and `alertmanager-discord-proxy` secrets,
11+
- creates/updates two Alertmanager integrations with fixed tokens (`alertmanager-critical`, `alertmanager-warning`),
12+
- configures outgoing webhooks (trigger: status change) to call the in-cluster Discord proxy.
13+
- If you rotate tokens/webhooks, update SOPS secrets and rerun the job:
14+
`k3s kubectl delete job/oncall-bootstrap -n monitoring-oncall`.
15+
16+
## Smoke test (FIRING → RESOLVED, 15s, no spam)
17+
This hits the OnCall Alertmanager integration endpoint directly (deterministic; good for formatting and “edit-on-resolve”).
18+
19+
1) Get integration URLs (from the node):
20+
```
21+
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
22+
critical=$(k3s kubectl -n monitoring get secret alertmanager-oncall -o jsonpath='{.data.critical_url}' | base64 -d)
23+
warning=$(k3s kubectl -n monitoring get secret alertmanager-oncall -o jsonpath='{.data.warning_url}' | base64 -d)
24+
```
25+
26+
2) Fire and resolve:
27+
```
28+
k3s kubectl -n monitoring-oncall run --rm -i alert-smoke --restart=Never --image=curlimages/curl -- sh -c '
29+
set -euo pipefail
30+
cat >/tmp/firing.json <<EOF
31+
{
32+
"receiver": "oncall-smoke",
33+
"status": "firing",
34+
"alerts": [
35+
{
36+
"status": "firing",
37+
"labels": {
38+
"alertname": "oncall-smoke-delay15",
39+
"severity": "warning",
40+
"cluster": "prod",
41+
"namespace": "monitoring",
42+
"instance": "51.158.67.237:9100"
43+
},
44+
"annotations": {
45+
"summary": "OnCall Discord smoke test",
46+
"description": "Expect one Discord message that updates on resolve"
47+
},
48+
"startsAt": "2025-01-01T00:00:00Z",
49+
"endsAt": "0001-01-01T00:00:00Z"
50+
}
51+
],
52+
"commonLabels": {
53+
"alertname": "oncall-smoke-delay15",
54+
"severity": "warning",
55+
"cluster": "prod",
56+
"namespace": "monitoring",
57+
"instance": "51.158.67.237:9100"
58+
},
59+
"commonAnnotations": {
60+
"summary": "OnCall Discord smoke test",
61+
"description": "Expect one Discord message that updates on resolve"
62+
},
63+
"version": "4"
64+
}
65+
EOF
66+
cat >/tmp/resolved.json <<EOF
67+
{
68+
"receiver": "oncall-smoke",
69+
"status": "resolved",
70+
"alerts": [
71+
{
72+
"status": "resolved",
73+
"labels": {
74+
"alertname": "oncall-smoke-delay15",
75+
"severity": "warning",
76+
"cluster": "prod",
77+
"namespace": "monitoring",
78+
"instance": "51.158.67.237:9100"
79+
},
80+
"annotations": {
81+
"summary": "OnCall Discord smoke test",
82+
"description": "Expect one Discord message that updates on resolve"
83+
},
84+
"startsAt": "2025-01-01T00:00:00Z",
85+
"endsAt": "2025-01-01T00:00:15Z"
86+
}
87+
],
88+
"commonLabels": {
89+
"alertname": "oncall-smoke-delay15",
90+
"severity": "warning",
91+
"cluster": "prod",
92+
"namespace": "monitoring",
93+
"instance": "51.158.67.237:9100"
94+
},
95+
"commonAnnotations": {
96+
"summary": "OnCall Discord smoke test",
97+
"description": "Expect one Discord message that updates on resolve"
98+
},
99+
"version": "4"
100+
}
101+
EOF
102+
curl -sS -XPOST -H "Content-Type: application/json" -d @/tmp/firing.json "$warning"
103+
sleep 15
104+
curl -sS -XPOST -H "Content-Type: application/json" -d @/tmp/resolved.json "$warning"
105+
'
106+
```
107+
108+
Expected:
109+
- Exactly one Discord message in `#alerts-warning` with the `@OnCall` mention.
110+
- The message is edited in-place on resolve (no second message).
111+
- The title links to the OnCall alert group.
112+
113+
## Logs
114+
- Discord proxy: `k3s kubectl -n monitoring-oncall logs deploy/oncall-discord-proxy --tail=200`
115+
- OnCall engine: `k3s kubectl -n monitoring-oncall logs deploy/oncall-engine --tail=200`
116+
- Alertmanager: `k3s kubectl -n monitoring logs sts/alertmanager-monitoring-kube-prometheus-alertmanager --tail=200`

‎docs/ci.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# CI (quality gate + generated files)
2+
3+
We use GitHub Actions to validate Terraform and to keep a couple of generated artifacts up to date.
4+
5+
## Triggers
6+
- `pull_request` to `env/prod` or `env/test`: run Terraform checks (read-only).
7+
- `push` to `env/prod` or `env/test`: run Terraform checks and, if needed, auto-commit generated files back to the same branch.
8+
9+
## Terraform checks (no external creds)
10+
Run in `terraform/`:
11+
- `terraform fmt -check -recursive`
12+
- `terraform init -backend=false -input=false`
13+
- `terraform validate -no-color`
14+
15+
## Generated by CI (do not edit manually)
16+
On pushes to `env/prod` / `env/test`, CI generates and commits:
17+
- `TERRAFORM.md` (via `terraform-docs`)
18+
- `KUBEDIAGRAM.png` (via `KubeDiagrams`, rendered from `terraform/k8s/apps/kustomization.yaml`)
19+
20+
The workflow uses only the default `GITHUB_TOKEN` (no extra secrets) with `contents: write` permission for the auto-commit step.
21+
22+
## Concurrency policy
23+
Only one run per target branch (`env/prod` or `env/test`) is allowed at a time to avoid concurrent pushes of regenerated outputs.
24+

‎docs/monitoring.md‎

Lines changed: 21 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,19 +6,11 @@ Provisioning
66

77
View current dashboards
88
- Grafana: `https://${ENV_PREFIX}monitoring.${BASE_DOMAIN}`.
9-
- Admin credentials (read from secret):
10-
```
11-
k3s kubectl -n monitoring-grafana get secret monitoring-grafana -o jsonpath='{.data.admin-user}' | base64 -d
12-
k3s kubectl -n monitoring-grafana get secret monitoring-grafana -o jsonpath='{.data.admin-password}' | base64 -d
13-
```
9+
- Admin credentials: see `docs/secrets.md` (“Get credentials from the cluster”).
1410

1511
Update existing dashboards
1612
- Edit the JSON in `terraform/k8s/grafana-dashboards/`.
17-
- Commit to `env/test`, fast-forward to `env/prod`, reconcile Flux:
18-
```
19-
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
20-
flux reconcile kustomization apps -n flux-system --with-source
21-
```
13+
- Commit to the branch and reconcile Flux (commands: `docs/getting-started.md`).
2214
- Grafana auto-reloads after rollout restart if needed:
2315
`k3s kubectl rollout restart deploy/grafana -n monitoring-grafana`
2416

@@ -35,3 +27,22 @@ Add a new dashboard
3527
Notes
3628
- Datasource is pre-provisioned to Prometheus (`monitoring-kube-prometheus-prometheus.monitoring.svc:9090`); dashboards should reference it as default.
3729
- If you remove a dashboard JSON from the repo, Grafana will drop it on next reconcile.
30+
31+
## Alerting (Alertmanager → OnCall → Discord)
32+
- See `docs/alerts.md` for the full flow and smoke tests.
33+
- Alert rules live in `terraform/k8s/monitoring-alerts.tpl.yaml` (node readiness, disk/pvc pressure, CoreDNS/control-plane targets, Flux stalled/failed, CrashLoop, HPA max, etc.).
34+
35+
## Image digest rollout (www/blog)
36+
- Flux image automation resources live in `terraform/k8s/image-automation.yaml` (ImageRepository/ImagePolicy/ImageUpdateAutomation) and can update manifests in-repo when images are published as immutable tags.
37+
- Until that is fully relied on for `www/blog`, CronJob `digest-rollout` (namespace `website`) runs every 2m:
38+
- reads latest digest for `www-website:latest` and `www-blog:latest`,
39+
- compares with deployment annotation,
40+
- if changed, patches the deployment annotation to force a restart.
41+
- Containers use `imagePullPolicy: Always`, so new pods pull the updated digest.
42+
- Run job manually:
43+
```
44+
k3s kubectl create job --from=cronjob/digest-rollout digest-rollout-manual -n website
45+
k3s kubectl logs job/digest-rollout-manual -n website
46+
k3s kubectl delete job/digest-rollout-manual -n website
47+
```
48+
- To verify rollout on new image: push `latest` to GHCR, wait ≤2m, then `k3s kubectl rollout status deploy/devsh-website -n website` (same for `devsh-blog`).

‎docs/resources.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Resource management (CPU/RAM) and uptime
2+
3+
This cluster is a single-node k3s setup, so one noisy workload can starve everything else. The goal is to keep user-facing services responsive (www/blog, Kimai) and make batch jobs “yield” under pressure.
4+
5+
## What we enforce
6+
- Requests/limits on key workloads to prevent unbounded RAM usage.
7+
- Priority classes so critical apps can preempt low-priority batch pods when the node is under pressure.
8+
- Rollout strategies that avoid briefly running duplicate heavy pods on a single node.
9+
10+
## Where it is defined
11+
- Priority classes: `terraform/k8s/priority-classes.tpl.yaml`
12+
- `prod-app-critical` (user-facing apps)
13+
- `prod-batch-low` (non-critical batch jobs)
14+
- Website/Blog deployments: `terraform/k8s/www-sites.tpl.yaml`
15+
- Kimai + MariaDB deployments: `terraform/k8s/kimai.tpl.yaml`
16+
- Batch CronJobs (example): `terraform/k8s/image-digest-rollout.yaml`
17+
18+
## How to check CPU/RAM usage
19+
Run on the node:
20+
```
21+
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
22+
```
23+
24+
Kubernetes view (requires metrics-server):
25+
```
26+
k3s kubectl top nodes
27+
k3s kubectl top pods -A --sort-by=memory
28+
```
29+
30+
Node view:
31+
```
32+
free -h
33+
ps aux --sort=-rss | head -n 20
34+
```
35+
36+
Recent evictions / OOM signals:
37+
```
38+
k3s kubectl get events -A --sort-by=.lastTimestamp | tail -n 80
39+
dmesg -T | rg -i "oom|killed process" || true
40+
```
41+
42+
## How to change resources safely
43+
- Keep limits realistic; on a single node, “too high” limits just move OOM pressure to the node.
44+
- Use `priorityClassName: prod-app-critical` only for user-facing services that must stay up.
45+
- For single replicas of stateful workloads, avoid rollouts that create duplicates. Prefer `maxSurge: 0` (or `Recreate`) and a sensible `terminationGracePeriodSeconds`.
46+

0 commit comments

Comments
 (0)