Terramate GitLab CI Template

This repository is the source of reusable GitLab CI templates for Terramate and Terragrunt.

The root pipeline in ./.gitlab-ci.yml is the template repository's own integration pipeline. It consumes the exported files locally so changes to the shared YAML are exercised in this repository before other repositories adopt them.

How this template is created

Most parts of the template is created using AI or AI assisted. Ideas, inputs and design decision are human controlled.

How this template is designed

This template is inspired by TACOS like Atlantis. In an ideal situation, use of TACOS like Atlantis, Digger is desirable. There are some use cases where the users want a pure GitLab CI solution for use with Terragrunt. At current stage, I think there still areas for improvement for this template. (e.g. DAG, resources dependency, plan/apply abort in mid-flight, how to resume, drifts due to mid-flight abort, etc.)

So, in this template, it would:

• Try to block users to run two Terragrunt with GitLab pipeline simultaneously (controlled by GitLab resource group)
• Two different users or Git branches dis-regard / overwrite existing changes that is already commited in main branch (controled/checked by check-branch-freshness)
• Applying Terragrunt should only apply a pre-defined Terraform plans. Instead of plan again right before applying.
• Users will be able to preview what would be changed (task 'plan-mr-preview') at merge request (MR).
  • Plain Terraform output is hard to read. Use of tf-summarize and tfplan2md to help users to identify what would be changed
  • The tfplan2md plan is posted as a single self-updating note on the MR (notes API), so reviewers see the plan without opening the job log (Atlantis-like review UX)
  • tf-summarize output is saved per stack as .tf-summarize.md and .tf-summarize.html artifacts under .terragrunt-plans/reports/
  • Additionally, iam-policy-autopilot derives advisory least-privilege IAM identity policies from each stack's terragrunt show -json plan. The pretty JSON per stack lands in the job log and is rendered into the same single MR plan note as a collapsed 🔐 IAM policies section (per-stack ```json blocks + a jq statement count + a hint for `Warnings` wildcard `Resource` statements; the `--pretty` JSON is saved as `.terragrunt-plans/reports/.iam-policies.json` (a compact `.iam-policies.compact.json` copy is kept alongside), and a `-- ` line after each `` keeps the closing tag in its own paragraph. All calls are best-effort (`|| true`), so a missing binary or tool failure never fails the plan. `plan-main` / `apply` do not generate or publish policies. Telemetry is disabled (see `DISABLE_IAM_POLICY_AUTOPILOT_TELEMETRY` under Common Overrides). The per-stack policies are always kept as artifacts; set `IAM_POLICY_AUTOPILOT_TO_MR_COMMENT: "false"` to skip appending them to the MR note (e.g. when many resources change and the generated policies would make the MR comment huge) — see Common Overrides. The pretty JSON is also echoed to the job log (the job trace); set `IAM_POLICY_AUTOPILOT_APPEND_TO_JOB_LOG: "false"` to keep the log small on large diffs — see Common Overrides.
• If a MR looks good, users can merge the code to the main branch
  • A new pipeline will be created
    • Syntax checking will be run again
    • Terragrunt/Terraform planning (at task 'plan-main'), the Terraform plan will be encrypted and saved as GitLab artifacts
    • If the user wants to apply the Terragrunt code, he/she needs to run the apply task manually in GitLab
    • After the apply task has run successfully, the Terraform plans (artifacts) would be deleted to prevent them to be re-run (accidentially)
• Other aspects
  • Use of Terramate to detect what resources is being changed
  • Stacks with Terragrunt dependency blocks are planned and applied in dependency (DAG) order: if stack A changes, stacks that depend on A are re-planned too, and dependencies are always applied before dependents. New stacks that depend on not-yet-existing stacks must declare mock_outputs + mock_outputs_allowed_terraform_commands in their dependency blocks (see design/iac-plan-apply.md).
  • Security checking
  • I am using a self signed CA certificate for my homelabe environment. So, there would be settings for trusting the self signed CA certificate.

What configuration is required

• Users need to onboard Terramate with their Terragrunt setting for each Terragrunt directory (needed for existing and new resources in the future)
• GitLab pipeline
  • Refer below for the variables needed

Known behaviour or known problems

• After the user merges new MR to main, a new pipeline would start. In that new pipeline:

  • The user needs to review the plan in the plan-main task
  • The user needs to run the apply task manually
  • There is a chance that the user forgets to run the apply task
  • If another user (or the same user) check out the most update code from the main branch
    • It is allowed for the new code in the new branch / new code to merge to main branch again using a new MR
    • Also, it is allowed for the user to run the apply task after the latest MR is merged to main branch
      • In this case, the latest pipeline should include the changes of resources that have not been applied in the last pipeline
    • If the user goes back to the previous pipeline to run the apply task, it should block
    • It would block because the previous task's git commit id/tip is not pointing to the latest one

• Suppose the plan passed the plan stage. MR is merged. A new task is spawn to work on the main branch. But the pipeline failed in the middle before it can be applied. i.e. the change in the IaC code is not applied to the related resources. Another MR is created to fix the problem, but this template/design may not be able to detect/aware that there are resource changes that had not been applied. This will cause drifting.

  • Users need to cater for these exception
    • e.g. Force the changed code to go thorugh the pipeine again or
    • Revert the changed code

Exported Include Files

• The templates are located in the templates/ directory and are designed to be included in other GitLab projects. Detailed documentation for each template can be found in the design/ folder:

• Base Configurations & CA Init - Global variables and TLS normalization.
• Security Scanning - Linting (TFLint, KICS and so on.) and reporting.
• Deployment Workflow - Encryption of plan files, MR reporting, and manual applies.
• Approval Gate - CE-compatible MR approval enforcement from a centrally-governed approver pool.
• Debug Vault Helpers - Authentication and secret retrieval with Hashicorp Vault, for debugging only.
• DefectDojo Integration - Upload reports to Defectdojo for centralized vulnerability management.
• Homelab Defaults - Specifically designed for a homelab setting. Do not include this in other environments. Set your own values via CI/CD variables instead.

Future Consumer Pattern

Future caller repositories should consume these files with include:project, pinned to a tag or commit SHA.

Example:

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH == "main"'
    - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
      when: never

stages:
  - syntax-check
  - plan
  - security-check
  - approval-gate
  - security-check-to-sast
  - upload-reports-to-defect-dojo
  - apply

include:
  - project: 'homelab/tg-s3-gl-terramate-template'
    # Update the branch/tag in next line
    ref: 'v1.0.0'
    file:
      - 'templates/iac-security.yml'
      - 'templates/iac-plan-apply.yml'
      - 'templates/iac-approval-gate.yml'
      - 'templates/upload-reports-to-defectdojo.yml'
      # Optional: include only if you use SOPS AGE / Hashicorp Vault for secret retrieval
      # - 'templates/debug-vault-helpers.yml'

The caller repository keeps ownership of its Terramate stack definitions, Terragrunt backend configuration, Terraform/OpenTofu code, and project-level CI/CD variables.

IaC Approval Gate (CE)

templates/iac-approval-gate.yml enforces merge-request approvals on GitLab CE (no Premium features): the gate jobs count the MR's approvals from a centrally-governed approver pool (a separate iac-approvers repo) and fail closed unless the required threshold is met — blocking the merge via "Pipelines must succeed" and blocking apply via an optional needs entry. Security scanners are advisory (allow_failure: true); the approval gate is the merge blocker. See design/iac-approval-gate.md for wiring, variables, tokens, the project-settings checklist and the threat model.

Required CI/CD Variables

• IAC_PIPELINE_IMAGE — container image to use for plan/apply jobs
  • You need to have these tools installaed in the IaC pipeline container
    • Terragrunt
    • Terraform/OpenTofu - I tested with OpenTofu
    • Terramate
    • openssl - For encrypting Terraform state as GitLab artifact
    • tfsummarize
    • tfplan2md
    • glow
    • iam-policy-autopilot - advisory least-privilege IAM policy generation from Terraform plans (MR preview only). Expected at /usr/local/bin/iam-policy-autopilot
• LOCAL_INTERMEDIATE_CA_CERT_FILE
• LOCAL_ROOT_CA_CERT_FILE
• TF_STATE_ARTIFACT_ENCRYPTION_KEY
• GITLAB_API_TOKEN — Project or group access token with api scope and Maintainer/Owner role. Used by plan-mr-preview to post the MR plan note, by plan-main to post the plan as a commit comment, and by the apply job to delete plan-main artifacts after a successful apply to prevent replay. Add it as a masked CI/CD variable and leave it unprotected so MR pipelines from non-protected branches can read it. If unset, note posting and artifact deletion are skipped (non-fatal). on-fatal).

Optional: AWS IAM Roles Anywhere (RA)

The template supports AWS IAM Roles Anywhere authentication so the GitLab runner can assume an IAM role using an X.509 client certificate signed by your own on-prem CA — no static AWS access keys. Two pieces are required:

1. In the consumer repo's .gitlab-ci.yml — set the flag and non-secret identifiers:

variables:
  AWS_RA_ENABLED: "true"
  AWS_RA_REGION: "us-east-2"
  AWS_RA_TRUST_ANCHOR_ARN: "arn:aws:rolesanywhere:..."
  AWS_RA_PROFILE_ARN: "arn:aws:rolesanywhere:..."
  AWS_RA_ROLE_ARN: "arn:aws:iam::...:role/..."

2. In GitLab Settings → CI/CD → Variables — set the secret material:

Variable	Type	Protected	Masked	Value
AWS_RA_CLIENT_CERT_FILE	File	✅	—	Runner x509 client cert (PEM)
AWS_RA_CLIENT_KEY_BASE64	Variable	✅	✅	base64 -w0 of the matching private key

The key is provided as base64 because a multi-line PEM private key cannot be masked as a GitLab CI/CD variable. The template decodes it at runtime (base64 -d). The plan/apply jobs will fail fast with clear error messages if any AWS_RA_* variable is set but incomplete.

Optional CI/CD Variables

• SOPS_AGE_KEY or SOPS_AGE_KEY_FILE
• TFLINT_PLUGIN_GITHUB_TOKEN
• DEFECTDOJO_URL
• DEFECTDOJO_API_KEY
• DEFECTDOJO_ENGAGEMENT_ID
• VAULT_ADDR — required when using Vault helper jobs
• VAULT_ID_TOKEN_AUDIENCE — required when using Vault helper jobs
• VAULT_AUTH_ROLE — required when using Vault helper jobs
• IAC_APPROVERS_REPO_URL — full https clone URL of the iac-approvers repo. Required when including templates/iac-approval-gate.yml — when empty the gate jobs fail closed (misconfiguration, not "off").
• IAC_APPROVERS_REF — branch or tag of the iac-approvers repo the gate clones (default main; recommend a pinned tag in real use).
• IAC_APPROVAL_REQUIRED — consumer default approval threshold (default 1). Ignored when a central required.txt exists in the approvers repo — central policy wins.
• IAC_APPROVAL_EXCLUDE_AUTHOR — true (default): the MR author's own approval never counts; false: it counts (solo-maintainer case).
• IAC_APPROVAL_GITLAB_TOKEN — approvals-API fallback token (read_api scope, Reporter role is sufficient), masked and unprotected. Only needed if the approvals endpoint rejects the job token.

Common Overrides

These shared defaults can be overridden by caller repositories without editing the template:

• IAC_PIPELINE_IMAGE — no default; must be set by the caller
• IAC_RESOURCE_GROUP
• AWS_EC2_METADATA_DISABLED — defaults to false
• VAULT_AUTH_PATH — defaults to jwt
• DISABLE_IAM_POLICY_AUTOPILOT_TELEMETRY — defaults to "true" (opt out of iam-policy-autopilot telemetry). The env var has the highest precedence — it beats the persisted ~/.iam-policy-autopilot/config.json choice. A best-effort iam-policy-autopilot telemetry --disable is also run in the plan job. Set to "false" only if you explicitly want telemetry.
• IAM_POLICY_AUTOPILOT_TO_MR_COMMENT — defaults to "true". The generated IAM policies are always kept as plan-mr-preview artifacts (.terragrunt-plans/reports/<stack>.iam-policies.json). Set to "false" to skip appending them to the MR plan note — useful when many resources change and the policies would make the MR comment very large.
• IAM_POLICY_AUTOPILOT_APPEND_TO_JOB_LOG — defaults to "true". The pretty JSON is also echoed to the plan-mr-preview job log (the job trace, not the MR comment). Set to "false" to keep the job log small when many resources change; the artifacts and the optional MR comment still exist.

Notes

• The shared plan/apply logic preserves the current encrypted saved-plan workflow.
• The optional upload include expects DefectDojo integration variables and should only be included by repositories that use DefectDojo.
• Future caller repositories should pin template consumption to a release tag or commit SHA rather than a moving branch.