Skip to content

examples/terraform/text-to-sql: a GKE cluster for the text-to-SQL recipe - #441

Open
linde wants to merge 1 commit into
openrl-dev:mainfrom
linde:terraform/text-to-sql
Open

linde wants to merge 1 commit into
openrl-dev:mainfrom
linde:terraform/text-to-sql

Conversation

@linde

@linde linde commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Terraform for a small GKE cluster that runs the text-to-SQL recipe as
published, on its own model, google/gemma-4-e2b: one g2-standard-24 node
with two NVIDIA L4 GPUs, one for the LoRA trainer and one for the vLLM
sampler. It is the GKE Setup Guide as code, kept to the minimum needed to
run the recipe (default network, default node service account), so it is not
a production setup.

It is two roots, applied in order. 1-infra enables the APIs and creates a
zonal GKE Standard cluster with the Filestore CSI driver and the GPU pool,
with GKE's own driver install and device plugin off so DRA can hand out the
GPUs (zone and gpu_spot are variables, since L4s stock out). 2-cluster is
separate because its Helm and kubectl providers can only be configured once
the cluster exists. It installs the NVIDIA driver, the NVIDIA DRA driver,
and the OpenRL LoRA control plane, and outputs the port-forward and the
command that runs the recipe against the cluster.

The control plane is the recipe's own overlay, examples/text-to-sql (the
LoRA bundle k8s/deploy/lora with Gemma 4 E2B as the default model and a
part-of: text-to-sql label), so it stays in sync with the checked-out commit
and there is no vendored copy to drift. The run_recipe output runs the
recipe's preset with no model override.

2-cluster renders the overlay at plan time with the third-party
kbst/kustomization provider (render only, it never talks to the cluster),
writes the result to charts/openrl/.tmp/openrl-lora.yaml, and installs it
through a local Helm chart with no logic: its only template prints that file
with .Files.Get. The file lives inside the chart because Helm can only read
files there, and is written with local_sensitive_file only to keep the ~600
KB bundle out of the plan. Helm does not notice a changed file in a local
chart, so the release carries the bundle's sha256 as an unused value; a
change to the overlay or under k8s/ then shows up in the next plan and the
apply upgrades the release, which also deletes objects removed from the
overlay. main.tf and the README call out the third-party provider.

The configs declare only the non-hashicorp provider sources and pin no
versions. .gitignore now ignores Terraform state, .terraform/, lock files,
tfvars (commit *.tfvars.example only) and plans, the recipe's artifacts/ run
outputs, local credentials, and any .tmp/ directory, which holds the
rendered bundle.

@linde
linde force-pushed the terraform/text-to-sql branch 3 times, most recently from da71b9d to 5354ef7 Compare November 9, 2026 23:56
Terraform for a small GKE cluster that runs the text-to-SQL recipe as
published, on its own model, google/gemma-4-e2b: one g2-standard-24 node
with two NVIDIA L4 GPUs, one for the LoRA trainer and one for the vLLM
sampler. It is the GKE Setup Guide as code, kept to the minimum needed to
run the recipe (default network, default node service account), so it is not
a production setup.

It is two roots, applied in order. 1-infra enables the APIs and creates a
zonal GKE Standard cluster with the Filestore CSI driver and the GPU pool,
with GKE's own driver install and device plugin off so DRA can hand out the
GPUs (zone and gpu_spot are variables, since L4s stock out). 2-cluster is
separate because its Helm and kubectl providers can only be configured once
the cluster exists. It installs the NVIDIA driver, the NVIDIA DRA driver,
and the OpenRL LoRA control plane, and outputs the port-forward and the
command that runs the recipe against the cluster.

The control plane is the recipe's own overlay, examples/text-to-sql (the
LoRA bundle k8s/deploy/lora with Gemma 4 E2B as the default model and a
part-of: text-to-sql label), so it stays in sync with the checked-out commit
and there is no vendored copy to drift. The run_recipe output runs the
recipe's preset with no model override.

2-cluster renders the overlay at plan time with the third-party
kbst/kustomization provider (render only, it never talks to the cluster),
writes the result to charts/openrl/.tmp/openrl-lora.yaml, and installs it
through a local Helm chart with no logic: its only template prints that file
with .Files.Get. The file lives inside the chart because Helm can only read
files there, and is written with local_sensitive_file only to keep the ~600
KB bundle out of the plan. Helm does not notice a changed file in a local
chart, so the release carries the bundle's sha256 as an unused value; a
change to the overlay or under k8s/ then shows up in the next plan and the
apply upgrades the release, which also deletes objects removed from the
overlay. main.tf and the README call out the third-party provider.

The configs declare only the non-hashicorp provider sources and pin no
versions. .gitignore now ignores Terraform state, .terraform/, lock files,
tfvars (commit *.tfvars.example only) and plans, the recipe's artifacts/ run
outputs, local credentials, and any .tmp/ directory, which holds the
rendered bundle.

Signed-off-by: Steve Linde <stevenlinde@google.com>
@linde
linde force-pushed the terraform/text-to-sql branch from 5354ef7 to 54e65c7 Compare November 18, 2026 17:12

data "kustomization_build" "openrl" {
path = "${path.module}/../../../../examples/text-to-sql"
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The raw kustomize build keeps api-server:latest and scheduler:latest (only make render VERSION=… sets tags), so images don't follow the checked-out commit. Can we set the tags here, or drop the README's "check out a release tag for a fixed deployment" line?

resource "google_container_cluster" "openrl" {
name = "open-rl-ttsql"
location = var.zone
initial_node_count = 0 # no default node pool; the GPU pool is below

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The guide has a CPU pool, here the control plane runs on the GPU node. If GKE adds the nvidia.com/gpu taint, the API server, scheduler and Redis have no toleration and stay Pending. Did a clean apply come up? If the node is tainted, a small CPU pool would fix it.


## Tear down

Destroy stage 2 first: removing the OpenRL bundle deletes its PVC, which deletes the Filestore instance before the cluster goes.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two risks here: 2. worker CRs keep their openrl.io/placement finalizer once the scheduler is gone, so the destroy may hang 2. the Filestore delete is async, so it can outlive the cluster.

@droot

droot commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Thanks for this. The Terraform layout is clean, and the two-root split is correct. My main concern is maintenance. Details follow, most important first.

1. Maintenance

Three paths now render the bundle. The release bundle and the guide use make render. This root uses kbst/kustomization. That path does not pin images or apply the storage options. The cluster runs :latest images, even on a release tag.

The root copies facts that change often. These are the DRA chart version and values, the driver installer URL, the node labels, the service name, port 9003, the PVC name and the preset name. The guide changed for these facts in #418, #253 and cac7e3c. Each future change must now go into two places.

Nothing tests the root. CI runs only ruff. Provider versions are not pinned, and the lock files are gitignored. When a provider or a name changes, the root breaks silently. A user finds the failure during apply.

There are four layers to debug. The path is: overlay, Terraform provider, local file, Helm chart, Helm Secret. A change that works with kubectl apply can fail here. An example is a custom resource with its CRD in the same bundle.

The third-party provider adds risk for an optional feature. The recipe README says that the release bundle runs the recipe without changes. The overlay only sets the default BASE_MODEL.

The scope will grow. The path examples/terraform/<recipe> invites copies for harvey-labs and autoresearch. Each copy repeats 1-infra and the driver steps.

2. Discovery and coexistence with the guide

  • No page links to this root. Seven pages link to gke-setup.md.
  • The name is too narrow. The root builds a general LoRA cluster on L4s, not only a text-to-SQL cluster.
  • The root and the guide are already different. The differences are zonal or regional, the release channel, the CPU pool, the DRA namespace, release bundle or checkout, and the cleanup steps. No document says which differences are intentional.
  • Do not mix the two on one cluster. Helm does not adopt resources that kubectl apply created. Also, a kubectl apply changes resources that Helm owns.
  • The cleanup steps are different. The guide deletes the cluster. This README destroys stage 2 first. If a user follows the wrong steps, a 1 TiB Filestore instance continues to bill.

3. Correctness (please check)

  • GPU taint. initial_node_count = 0 without remove_default_node_pool can leave an empty default-pool. Then GKE can taint the GPU node with nvidia.com/gpu=present:NoSchedule. The API server, scheduler and Redis have no toleration, so they stay Pending. Please add a CPU pool, as the guide does.
  • Teardown. Workload CRs have the finalizer openrl.io/placement. Only the scheduler removes it, and Helm deletes the scheduler at the same time. The CRs, the CRD and the namespace stay in Terminating. Worker pods keep the PVC, so the Filestore instance is not deleted. Then 1-infra destroy removes the cluster, and the Filestore instance stays.
  • Spot. With gpu_spot = true, a preemption also stops the control plane and Redis. Please document this.

Suggestions

  1. Make a general GKE root, for example deploy/terraform/gke, with a release variable. Install the release asset with data "http". This removes the third-party provider and the local chart, and it pins the images.
  2. Make the guide the reference. Add a link to the root at the top of gke-setup.md. List the intended differences in the root README.
  3. Add terraform fmt -check and terraform validate to CI. Pin the providers. Commit the lock files.
  4. Add a CODEOWNERS entry for the root, so that changes to the guide or to k8s/deploy/lora include its owner.
  5. Make the cleanup steps the same in both documents. Delete the Workload CRs first. Make sure that the Filestore instance is gone. Then delete the cluster.

If you keep the overlay, please do items 2 to 5 at least. Also, render through make render, so that the images are pinned.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants