011 - Infrastructure Code Disaggregation

[cpcundill]

Open Design Proposal - 011 - Infrastructure Code Disaggregation

Author(s)

• Chris Cundill @cpcundill

Introduction

Disaggregation of Infrastructure code is proposed in order to realise the following improvements:

• Isolation and simplification for future maintenance
• Reduced time to apply changes
• Improved concurrent working for Infrastructure Engineers

Status

OPEN

• Draft: proposal is still being authored and is not officially open for comment yet
• Open: proposal is open for comment
• Closed: proposal is closed for comment with implementation expected
• On Hold: proposal is on hold due to concerns raised/project changes with implementation not expected

Detail

Problem

Big ball of state

The Digital Land Infrastructure codebase, implemented with Terraform, has grown organically over time. Originally only responsible for provisioning the Platform system, it has now grown to provision AWS resources for the whole Digital Land Service which spans the Platform, Data Collection Pipeline, Providers and Data Design systems.

Terraform uses a state file to map infrastructure resources definitions (code) to real-world infrastructure resources (e.g. in AWS), i.e.

module.domain.zone (code definition) -> Z009331066ITDHC548SN8 (AWS Route 53 Zone ID)

State must be locked when applying changes. At the time of writing (Dec '24), the Terraform codebase currently manages over 1,300 AWS resources. The large size of the state has begun to be problematic for a few reasons:

• Only one engineer can apply changes to an environment at any one time
• Refreshing the state against code definitions is slower

Any a minor change currently requires the reading of state and AWS resources across the whole programme infrastructure and blocks any other change being applied concurrently. An innocuous change such as introducing a new CloudWatch Log metric filter should not block the rollout of an ECS cluster change - and vice-versa.

Borrowing terminology from the architecture abstractions encapsulated in the C4 modelling approach, the current infrastructure codebase is responsible for the whole system context of the programme. This is unsustainable.

Environment Branches

The current behaviour of the Terraform provisioning code expects all resources to be deployed into all environments. Granted, there is some customisation of parameters between environments, but not much in the way of control over which resources are included. Unfortunately, by necessity, this has led to maintaining an environment branch (dev) for the development environment. The main branch is used to represent the Terraform code for application to staging and production environments. Often, engineers are required to develop features against the dev branch and then reimplement against the main branch for promotion to higher environments.

Solution

Summary

Disaggregate the Terraform codebase into multiple units based around systems:

• Core
  • Responsible for the foundational elements of hosting infrastructure for programme, including networking, storage, DNS and ECS (container cluster)
• Platform
  • Responsible for the website and API to expose Digital Planning data
• Data Collection Pipeline
  • Responsible for extracting, transforming and loading Digital Planning data from providers into the Platform
• Providers
  • Responsible for the services offered to data providers, including the Submit tool
• Data Design
  • Responsible for designing and maintaining data standards for Digital Planning

Indicative code structure

The provisioning code for each system can exist within a sub-folder of the infrastructure code repository and have its own dedicated state; this will allow for the application of changes to different systems concurrently within the same environment. Existing modules can be shared by the systems and can undergo some refactoring and simplification as necessary.

Each system must be independent and use data lookups for any dependencies rather than code-level references.

Indicative repository folder structure

The top level structure within the code repository will resemble will include two main folders:

 ├── modules
 ├── systems

Modules will continue to be the container for shared modules while the systems folder will contain isolated units (each with its own independent state).

A more expanded view of the folder structure would reveal:

.
├── Makefile
├── README.md
├── modules
│   ├── airflow
│   ├── application
│   ├── application-task
│   ├── canaries
│   ├── domain
│   ├── ...
├── systems
│   ├── core
│   ├── platform
│   ├── data-collection-pipeline
│   ├── providers
│   ├── data-design
│   │   ├── environments
│   │   │   ├── development.tfvars
│   │   │   ├── staging.tfvars
│   │   │   └── production.tfvars
│   │   ├── README.md
│   │   ├── outputs.tf
│   │   ├── main.tf
│   │   ├── terraform.tf
│   │   └── variables.tf

Each system directory will contain an environments directory which holds Terraform values files for each environment into which the system is to be deployed. Care must be taken to ensure that the Terraform provisioning code within the system can handle no-op scenarios for a given environment. That is to say, it must be possible to deploy a system - and indeed aspects of a system - to selective environments. This principle is critical to solve the aforementioned environment branch problem and thus have a single branch (main) serving all environments. Any differences required between environments should be defined within environment-specific tfvar files.

Scalability

Should the Terraform state for an individual system become too large in future, that system could be further disaggregated into isolated abstractions. Borrowing again from C4 modelling terminology, the infrastructure resources for a system could be further organised into containers.

Implementation considerations

Migration of state

State in Terraform must be managed carefully since there is a mapping between code artifacts and AWS resources, almost all kinds of refactoring changes (file moves, renames) will require corresponding changes to Terraform state. See https://developer.hashicorp.com/terraform/cli/state/move for details.

This can be achieved in a number of ways:

• Direct editing of State Files (stored in S3 buckets)
• Terraform state mv command
• Move configuration
• Third party tools, e.g. terraform-state-split and terraform-state-mover

Direct editing of state files is rarely recommended and should be avoided where at all possible.

Further reading

• Hashicorp support article about spitting state files
• Terraform Tidy-Up: Refactoring Your Infrastructure with Grace

Migration Stages

The following sequential stages of migration are recommended:

• Stage 1: Branch consolidation
  • Consolidate of dev and main branches to provide simple baseline for refactoring
• Stage 2: Extract Data Design
  • Discrete small system which serves as a suitable pilot to trial state migration methods
• Stage 3: Core system
  • Core system - Domain
    • Start new core system and extract domain code
    • Revise dependent code to use data lookups for domain instead of resource references
  • Core system - Network
    • Extract network code into core system
    • Revise dependent code to use data lookups for network elements instead of resource references
  • Core system - Storage
    • Extract storage code into core system
    • Consider bringing the Async database under management of this system
    • Revise dependent code to use data lookups for network elements instead of resource references
• Stage 4: Providers
  • Moderately sized and includes a frontend
• Stage 5: Data Collection Pipeline
  • Includes definition of MWAA (Airflow) resources
• Stage 6: Platform
  • Apply learnings from previous migrations to move the largest system of the programme

Environment migration strategy

For each of the outlined stages, the Terraform code and state would applied in all environments as to ensure no delayed integration issues are encountered.

Design Comments/Questions

Leave comments and questions in this discussion.