Create pipeline to handle image promotion from private registry to public

[sjonpaulbrown]

Create Pipeline for Image Promotion

This PR aims to streamline the build and deployment process for Docker images by introducing workflows that seamlessly manage builds in a private container registry and promote those images to a public registry. The changes provide greater flexibility in managing individual promotions while ensuring private builds remain efficient and autonomous.

Overarching Goals

1. Streamlined Builds and Deployments:

   • Introduce workflows to facilitate the promotion of specific roles or environments to a public registry as needed.

2. Flexible Image Promotions:

   • Allow the flow-protocol-release team to manage and approve the promotion of individual images to the public registry for each environment.
   • Provide granular control over which images are promoted, ensuring only validated builds are made public.

3. Autonomous Private Builds:

   • Ensure that private builds are completed independently and do not require approval or additional manual steps.
   • Use secure authentication and environment-specific configurations to handle private registry access seamlessly.

4. Service Account Separation for Security:

   • Distinct service accounts are used for interacting with the private and public registries:
     • A private registry service account is used exclusively for writing images to the private container registry.
     • A separate public registry service account is used for both reading from the private registry and writing to the public registry during promotions.
   • This separation enhances security by adhering to the principle of least privilege, ensuring each service account has only the permissions required for its specific task.

Key Features

• Private Registry Builds:

  • New private-build job to handle private Docker image builds for various roles (access, collection, consensus, execution, observer, verification).
  • Environment variable-driven configurations for registry URLs, credentials, and tags.
  • Supports matrix-based parallel builds for faster execution.

• Public Registry Promotions:

  • Individual promotion jobs for each role, allowing precise control over what gets published to the public registry.
  • The flow-protocol-release team is granted access to manage these promotions, ensuring proper governance and quality assurance.

• Simplified Workflows:

  • Updated workflow to support both private and public image deployments

cd.yml Workflow Overview

1. Private Build Process:

   • Developers trigger builds using the workflow_dispatch event or other defined inputs.
   • Images are built and pushed to the private registry using the private registry service account, without requiring manual approval.
   • Role-specific images and tags are handled as per input configurations.

2. Public Promotion Process:

   • The flow-protocol-release team has access to trigger promotion workflows for individual roles or environments.
   • The public registry service account reads images from the private registry, retags them, and pushes them to the public registry.
   • Promotions are tracked and managed independently for each role, ensuring flexibility and control.

builds.yml Workflow Overview

1. The builds.yml workflow has been modified to strictly use the private registry

• The optional input for private builds has been removed
• The authentication to the public registry has been removed
• The secret & environment used to authenticate to the registry has been changed

Benefits

• Efficiency: Automates private builds and promotions to reduce manual effort.
• Flexibility: Empowers the flow-protocol-release team to manage public image promotions selectively.
• Governance: Ensures that only approved and validated images are promoted to the public registry.
• Security:
  • Clear separation of service accounts for private and public registries minimizes risks.
  • Strict authentication for both registries ensures secure image management.

Notes

• Ensure the necessary secrets (PRIVATE_REGISTRY_UPLOAD_SECRET, PUBLIC_REGISTRY_PROMOTION_SECRET) are set up in the repository.
• Document the promotion process for the flow-protocol-release team to facilitate smooth operation.

By implementing these changes, the build and deployment pipeline will become more streamlined, with enhanced security through service account separation and greater control over image promotions.

Documentation

The following documentation has been created to ensure internal users understand the process

• Technical Docs
• User Docs

Example Execution

• An example Execution of building & promoting a single image can be found here

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 41.00%. Comparing base (aedb8dc) to head (e5b56bd).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6826      +/-   ##
==========================================
- Coverage   41.00%   41.00%   -0.01%     
==========================================
  Files        2103     2103              
  Lines      184990   184990              
==========================================
- Hits        75862    75855       -7     
- Misses     102764   102766       +2     
- Partials     6364     6369       +5     

Flag	Coverage Δ
unittests	41.00% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.