Repository navigation
chore(deploy): mirror the 2026-10-24 backend-sync costcut into deploy config - #23550
Merged
Merged
Conversation
… config The T4 deploy (646e54d, run 40138604325) re-asserted backend-sync at 4Gi / max 80: #23495 mirrored the 05:54Z right-sizing, not the evening costcut. The config-train's service-level maxScale (80, from prod.yaml) also outranks the lead's template-level 50 in the deploy's live-resolve step, so the cap reverted on every deploy. - gcp_backend.yml: backend_sync_memory prod fallback 4Gi -> 6Gi (BACKEND_SYNC_MEMORY unset in every scope; the fallback deploys). - deploy-backend-stack: prod backend now pins its own live shape (cpu 2 / 8Gi / concurrency 20 / min 3 / max 100, prod-scoped; dev inherits). 8Gi needs >=2 CPU (#21668). - ops/prod.yaml: backend-sync max 80 -> 50, min 1 -> 5; backend-integration min 1 -> 3 (stops fighting the deploy flag). - staging deploy.sh: sync memory 4Gi -> 6Gi (EQUAL_SCALE_FIELDS parity). - runbook: prod default memory documented as 6Gi. - Contract pins: test_cloud_run_rightsizing_pins.py (new) holds the workflow fallback, deploy flags, ops values and staging parity; drift/render pins updated to the costcut values.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Mirrors the lead's 2026-10-24 evening costcut (the live right-sizing #23495 did not cover) into the deploy config, so the next train re-asserts it instead of reverting it. The T4 deploy (image 646e54d, run 40138604325) came back with backend-sync at 4Gi / max 80; the lead re-applied 6Gi / min 5 / max 50 live (22:45-22:48Z), and the config-train's stale service-level annotations were still fighting it tonight (drift evidence below).
Why cfgmirror (#23495) missed it
prod.yaml backend-sync.max_instances: 80"until the lead names the flip". The config-train writes the service-levelrun.googleapis.com/maxScalefrom that file, and the deploy'sResolve live sync maxScaleprefers the service-level annotation over the serving revision's template — so the lead's template-level 50 loses to the stale service-level 80 on every deploy. Verified live tonight: serving template 5/50, service-level 80.prod && 3); prod.yaml asserted 1, so the config-train kept idling the projection pump between applies.How
gcp_backend.yml:backend_sync_memoryprod fallback4Gi -> 6Gi(vars.BACKEND_SYNC_MEMORYverified unset in org, prod and development scopes; the fallback is what deploys).deploy-backend-stack: the backend step gains prod-scoped shape flags--cpu=2 --memory=8Gi --concurrency=20 --min-instances=3 --max-instances=100(dev branch empty — dev keeps inheriting). Values are the verified live serving revisionbackend-646e54d-40138604325-1(22:39Z). cpu=2 pairs with 8Gi per [loop-c] main CI red: gha-ERROR: (gcloud.run.deploy) spec.template.spec.containers[N].resources.limits.memory: Invalid value specified for contain #21668 (1-CPU 8Gi deploys are rejected outright).backend/deploy/ops/prod.yaml:backend-syncmax80 -> 50, min1 -> 5(the warm floor the conversation-finalization queue comment already sizes against: min 5 x concurrency 20);backend-integrationmin1 -> 3, equal to the deploy flag so the ops file never fights the deploy.backend/deploy/staging/deploy.sh: staging sync memory4Gi -> 6Gi(drift_checkEQUAL_SCALE_FIELDSparity).backend/docs/runbooks/sync-ingest-archive.md: prod default memory and workflow fallback documented as 6Gi; the memory-trial recipe stays generic.backend/tests/unit/test_cloud_run_rightsizing_pins.pyholds the workflow fallback literal, the deploy-flag shape, the ops values (including the redis_guard backfill bounds) and staging-vs-prod parity throughdrift_check.source_model. Updated pins:test_drift_check.py(6Gi) andtest_render_backend_runtime_env.py(backend legitimately carries--cpu=2now; every integration-specific flag still excluded from it, and backend's own prod-scoped shape is pinned literally).Verification
backend/test.sh+BACKEND_UNIT_TEST_FILE_LISTover the changed test files..github/scripts/check_ci_budget.pypasses with no ci-budget.yaml change (value/flags-expression edit only; no trigger, timeout, runner or job-shape change).drift_check.py source --env prodagainst a fresh ro-prod snapshot (22:5xZ): the only scale findings are the intended reconciliations —backend-sync service_min 1 != ops 5,backend-sync service_max 80 != ops 50,backend-integration service_min 1 != ops 3(the stale live service-level annotations the config-train will overwrite from this file). No other scale drift; the deploy-flag model matches live for all four services.Not changed (deliberate)
Resolve live sync maxScale); the cap's source of record is the ops file.Product invariants affected
.github/workflows/gcp_backend.yml(one deploy parameter value; no trigger, timeout, runner or new-workflow change).