Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Refactor Dockerfile & entrypoint #8923

Open
wants to merge 59 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
64be0ae
Refactor formatting & docs
upbqdn Oct 2, 2024
a248b14
Refactor the `runtime` stage in Dockerfile
upbqdn Oct 2, 2024
ca1620c
Remove unused code from `entrypoint.sh`
upbqdn Oct 9, 2024
ea8b119
Simplify `entrypoint.sh` setup
upbqdn Oct 9, 2024
7cba8cf
Revise docs & formatting
upbqdn Oct 9, 2024
56c65e6
Adjust default values for env vars
upbqdn Oct 9, 2024
be38132
Bump Rust v from 1.79 to 1.81 in Dockerfile
upbqdn Oct 9, 2024
0492b7a
Refactor `entrypoint.sh`
upbqdn Oct 10, 2024
6595740
Refactor `Dockerfile`
upbqdn Oct 10, 2024
22dc738
Add TODOs for monitoring stage to Dockerfile
upbqdn Oct 10, 2024
962d4d3
Merge branch 'main' into docker-refactor
upbqdn Oct 10, 2024
621754b
Refactor `Dockerfile`
upbqdn Oct 10, 2024
6b68592
Add TODOs for monitoring stage to Dockerfile
upbqdn Oct 10, 2024
c788ccc
Merge branch 'docker-refactor' of github.com:ZcashFoundation/zebra in…
upbqdn Oct 10, 2024
38837e4
Fix a typo
upbqdn Oct 10, 2024
b58a602
Merge branch 'main' into docker-refactor
upbqdn Oct 10, 2024
2ada296
Allow running `zebrad` in test mode
upbqdn Oct 11, 2024
99cd18f
Merge branch 'docker-refactor' of github.com:ZcashFoundation/zebra in…
upbqdn Oct 11, 2024
c718609
Merge branch 'main' into docker-refactor
upbqdn Oct 11, 2024
69b03d4
Allow custom config for `zebrad` in test mode
upbqdn Oct 11, 2024
6932d9a
Remove `curl` from the `runtime` Docker image
upbqdn Oct 11, 2024
6fe460d
Remove redundant echos
upbqdn Oct 11, 2024
c5010b8
Remove a malfunctioning CD test
upbqdn Oct 12, 2024
e05df78
Remove a redundant CI test
upbqdn Oct 12, 2024
e9f0479
Remove all packages from the `runtime` stage
upbqdn Oct 12, 2024
7422ecf
Merge branch 'main' into docker-refactor
upbqdn Oct 14, 2024
4fa064c
Docs cosmetics
upbqdn Oct 14, 2024
afeb05f
Merge branch 'main' into docker-refactor
upbqdn Oct 18, 2024
2a72a07
Merge branch 'main' into docker-refactor
upbqdn Jan 10, 2025
41c595b
Merge branch 'main' into docker-refactor
upbqdn Jan 29, 2025
110dde2
Clarify docs
upbqdn Jan 29, 2025
cc4f225
Bump Rust version
upbqdn Jan 29, 2025
f0fa369
Remove a security note
upbqdn Jan 29, 2025
195c84e
Explicitly specify network cache dir
upbqdn Jan 29, 2025
d4e951f
Explicitly specify cookie dir
upbqdn Jan 29, 2025
d842ff3
Set UID, GID and home dir for the `zebra` user
upbqdn Jan 29, 2025
e0e2048
Set a working dir for the `zebra` user
upbqdn Jan 29, 2025
e2abaf4
Don't remove `FEATURES`
upbqdn Jan 29, 2025
58f7b00
Try re-introducing the `testnet-conf` check
upbqdn Jan 30, 2025
3d47f03
`ZEBRA_CACHED_STATE_DIR` -> `ZEBRA_CACHE_DIR`
upbqdn Jan 30, 2025
f620589
Refactor the dir structure
upbqdn Jan 30, 2025
7495ef3
Check that `ZEBRA_CONF_PATH` exists in the image
upbqdn Jan 30, 2025
ecdf536
Improve the check for `ZEBRA_CONF_PATH`
upbqdn Jan 30, 2025
6230caa
Use different flag in the `ZEBRA_CONF_PATH` check
upbqdn Jan 30, 2025
c26e1ae
Simplify the `ZEBRA_CONF_PATH` check
upbqdn Jan 30, 2025
faf56a5
Fix spelling
upbqdn Jan 30, 2025
e6e0187
Comment out the `testnet-conf` CI check
upbqdn Jan 30, 2025
c846f2c
Add commented out `test-zebra-conf-path` CI check
upbqdn Jan 30, 2025
9a624c0
Reintroduce `testnet-conf` CI check
upbqdn Jan 31, 2025
0c858b3
Update the `custom-conf` CI check
upbqdn Jan 31, 2025
8500aa6
Add `v2.1.0.toml` conf file
upbqdn Jan 31, 2025
26b50fb
Refine the `v2.1.0.toml` conf file
upbqdn Jan 31, 2025
dae9b90
Remove `ZEBRA_LISTEN_ADDR` from the entrypoint
upbqdn Jan 31, 2025
8de6a03
Remove `ZEBRA_CHECKPOINT_SYNC` from the entrypoint
upbqdn Jan 31, 2025
14f5469
Stop supporting configuration of the RPC port
upbqdn Feb 1, 2025
6b8cfa4
Add default conf file
upbqdn Feb 1, 2025
984c231
Prepare Zebra's config in the entrypoint script
upbqdn Feb 5, 2025
debd99d
Merge branch 'main' into docker-refactor
upbqdn Feb 5, 2025
2d27af7
Remove unneeded packages from the `deps` target
upbqdn Feb 5, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,4 @@
!zebra-*
!zebrad
!docker/entrypoint.sh
!docker/default_zebra_config.toml
147 changes: 80 additions & 67 deletions .github/workflows/cd-deploy-nodes-gcp.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Google Cloud node deployments and tests that run when Rust code or dependencies are modified,
# but only on PRs from the ZcashFoundation/zebra repository.
# (External PRs are tested/deployed by GitHub's Merge Queue.)
# (External PRs are tested/deployed by GitHub's Merge Queue.)
#
# 1. `versioning`: Extracts the major version from the release semver. Useful for segregating instances based on major versions.
# 2. `build`: Builds a Docker image named `zebrad` with the necessary tags derived from Git.
Expand Down Expand Up @@ -28,85 +28,85 @@ concurrency:

on:
merge_group:
types: [ checks_requested ]
types: [checks_requested]

workflow_dispatch:
inputs:
network:
default: Mainnet
description: 'Network to deploy: Mainnet or Testnet'
description: "Network to deploy: Mainnet or Testnet"
required: true
type: choice
options:
- Mainnet
- Testnet
cached_disk_type:
default: tip
description: 'Type of cached disk to use'
description: "Type of cached disk to use"
required: true
type: choice
options:
- tip
- checkpoint
prefer_main_cached_state:
default: false
description: 'Prefer cached state from the main branch'
description: "Prefer cached state from the main branch"
required: false
type: boolean
need_cached_disk:
default: true
description: 'Use a cached state disk'
description: "Use a cached state disk"
required: false
type: boolean
no_cache:
description: 'Disable the Docker cache for this build'
description: "Disable the Docker cache for this build"
required: false
type: boolean
default: false
log_file:
default: ''
description: 'Log to a file path rather than standard output'
default: ""
description: "Log to a file path rather than standard output"

push:
# Skip main branch updates where Rust code and dependencies aren't modified.
branches:
- main
paths:
# code and tests
- '**/*.rs'
# hard-coded checkpoints and proptest regressions
- '**/*.txt'
# dependencies
- '**/Cargo.toml'
- '**/Cargo.lock'
# configuration files
- '.cargo/config.toml'
- '**/clippy.toml'
# workflow definitions
- 'docker/**'
- '.dockerignore'
- '.github/workflows/cd-deploy-nodes-gcp.yml'
- '.github/workflows/sub-build-docker-image.yml'
# Skip main branch updates where Rust code and dependencies aren't modified.
branches:
- main
paths:
# code and tests
- "**/*.rs"
# hard-coded checkpoints and proptest regressions
- "**/*.txt"
# dependencies
- "**/Cargo.toml"
- "**/Cargo.lock"
# configuration files
- ".cargo/config.toml"
- "**/clippy.toml"
# workflow definitions
- "docker/**"
- ".dockerignore"
- ".github/workflows/cd-deploy-nodes-gcp.yml"
- ".github/workflows/sub-build-docker-image.yml"

# Only runs the Docker image tests, doesn't deploy any instances
pull_request:
# Skip PRs where Rust code and dependencies aren't modified.
paths:
# code and tests
- '**/*.rs'
- "**/*.rs"
# hard-coded checkpoints and proptest regressions
- '**/*.txt'
- "**/*.txt"
# dependencies
- '**/Cargo.toml'
- '**/Cargo.lock'
- "**/Cargo.toml"
- "**/Cargo.lock"
# configuration files
- '.cargo/config.toml'
- '**/clippy.toml'
- ".cargo/config.toml"
- "**/clippy.toml"
# workflow definitions
- 'docker/**'
- '.dockerignore'
- '.github/workflows/cd-deploy-nodes-gcp.yml'
- '.github/workflows/sub-build-docker-image.yml'
- "docker/**"
- ".dockerignore"
- ".github/workflows/cd-deploy-nodes-gcp.yml"
- ".github/workflows/sub-build-docker-image.yml"

release:
types:
Expand Down Expand Up @@ -160,6 +160,24 @@ jobs:
disk_suffix: ${{ inputs.cached_disk_type || 'tip' }}
prefer_main_cached_state: ${{ inputs.prefer_main_cached_state || (github.event_name == 'push' && github.ref_name == 'main' && true) || false }}

# Test that Zebra works using $ZEBRA_CONF_PATH config
#
# TODO: Fix the bug in the test below.
#
# The bug is that setting `ZEBRA_CONF_PATH="/etc/zebrad/zebrad.toml"` doesn't
# work because that path doesn't point to any config file in the container.
#
# test-zebra-conf-path:
# name: Test CD custom Docker config file
# needs: build
# uses: ./.github/workflows/sub-test-zebra-config.yml
# with:
# test_id: "custom-conf"
# docker_image: ${{ vars.GAR_BASE }}/zebrad@${{ needs.build.outputs.image_digest }}
# grep_patterns: '-e "loaded zebrad config.*config_path.*=.*v1.0.0-rc.2.toml"'
# test_variables: '-e NETWORK -e ZEBRA_CONF_PATH="zebrad/tests/common/configs/v1.0.0-rc.2.toml"'
# network: ${{ inputs.network || vars.ZCASH_NETWORK }}

# Each time this workflow is executed, a build will be triggered to create a new image
# with the corresponding tags using information from Git
#
Expand All @@ -183,11 +201,11 @@ jobs:
needs: build
uses: ./.github/workflows/sub-test-zebra-config.yml
with:
test_id: 'default-conf'
test_id: "default-conf"
docker_image: ${{ vars.GAR_BASE }}/zebrad@${{ needs.build.outputs.image_digest }}
grep_patterns: '-e "net.*=.*Main.*estimated progress to chain tip.*BeforeOverwinter"'
test_variables: '-e NETWORK'
network: 'Mainnet'
test_variables: "-e NETWORK"
network: "Mainnet"

# Test reconfiguring the docker image for testnet.
test-configuration-file-testnet:
Expand All @@ -196,23 +214,11 @@ jobs:
# Make sure Zebra can sync the genesis block on testnet
uses: ./.github/workflows/sub-test-zebra-config.yml
with:
test_id: 'testnet-conf'
test_id: "testnet-conf"
docker_image: ${{ vars.GAR_BASE }}/zebrad@${{ needs.build.outputs.image_digest }}
grep_patterns: '-e "net.*=.*Test.*estimated progress to chain tip.*Genesis" -e "net.*=.*Test.*estimated progress to chain tip.*BeforeOverwinter"'
test_variables: '-e NETWORK'
network: 'Testnet'

# Test that Zebra works using $ZEBRA_CONF_PATH config
test-zebra-conf-path:
name: Test CD custom Docker config file
needs: build
uses: ./.github/workflows/sub-test-zebra-config.yml
with:
test_id: 'custom-conf'
docker_image: ${{ vars.GAR_BASE }}/zebrad@${{ needs.build.outputs.image_digest }}
grep_patterns: '-e "loaded zebrad config.*config_path.*=.*v1.0.0-rc.2.toml"'
test_variables: '-e NETWORK -e ZEBRA_CONF_PATH="zebrad/tests/common/configs/v1.0.0-rc.2.toml"'
network: ${{ inputs.network || vars.ZCASH_NETWORK }}
test_variables: "-e NETWORK"
network: "Testnet"

# Deploy Managed Instance Groups (MiGs) for Mainnet and Testnet,
# with one node in the configured GCP region.
Expand All @@ -234,15 +240,22 @@ jobs:
matrix:
network: [Mainnet, Testnet]
name: Deploy ${{ matrix.network }} nodes
needs: [ build, versioning, test-configuration-file, test-zebra-conf-path, get-disk-name ]
needs:
[
build,
versioning,
test-configuration-file,
test-zebra-conf-path,
get-disk-name,
]
runs-on: ubuntu-latest
timeout-minutes: 60
env:
CACHED_DISK_NAME: ${{ needs.get-disk-name.outputs.cached_disk_name }}
environment: ${{ github.event_name == 'release' && 'prod' || 'dev' }}
permissions:
contents: 'read'
id-token: 'write'
contents: "read"
id-token: "write"
if: ${{ !cancelled() && !failure() && ((github.event_name == 'push' && github.ref_name == 'main') || github.event_name == 'release') }}

steps:
Expand Down Expand Up @@ -271,8 +284,8 @@ jobs:
id: auth
uses: google-github-actions/[email protected]
with:
workload_identity_provider: '${{ vars.GCP_WIF }}'
service_account: '${{ vars.GCP_DEPLOYMENTS_SA }}'
workload_identity_provider: "${{ vars.GCP_WIF }}"
service_account: "${{ vars.GCP_DEPLOYMENTS_SA }}"

- name: Set up Cloud SDK
uses: google-github-actions/[email protected]
Expand Down Expand Up @@ -349,14 +362,14 @@ jobs:
# Note: this instances are not automatically replaced or deleted
deploy-instance:
name: Deploy single ${{ inputs.network }} instance
needs: [ build, test-configuration-file, test-zebra-conf-path, get-disk-name ]
needs: [build, test-configuration-file, test-zebra-conf-path, get-disk-name]
runs-on: ubuntu-latest
timeout-minutes: 30
env:
CACHED_DISK_NAME: ${{ needs.get-disk-name.outputs.cached_disk_name }}
permissions:
contents: 'read'
id-token: 'write'
contents: "read"
id-token: "write"
# Run even if we don't need a cached disk, but only when triggered by a workflow_dispatch
if: ${{ !failure() && github.event_name == 'workflow_dispatch' }}

Expand Down Expand Up @@ -386,8 +399,8 @@ jobs:
id: auth
uses: google-github-actions/[email protected]
with:
workload_identity_provider: '${{ vars.GCP_WIF }}'
service_account: '${{ vars.GCP_DEPLOYMENTS_SA }}'
workload_identity_provider: "${{ vars.GCP_WIF }}"
service_account: "${{ vars.GCP_DEPLOYMENTS_SA }}"

- name: Set up Cloud SDK
uses: google-github-actions/[email protected]
Expand Down Expand Up @@ -428,7 +441,7 @@ jobs:
failure-issue:
name: Open or update issues for release failures
# When a new job is added to this workflow, add it to this list.
needs: [ versioning, build, deploy-nodes, deploy-instance ]
needs: [versioning, build, deploy-nodes, deploy-instance]
# Only open tickets for failed or cancelled jobs that are not coming from PRs.
# (PR statuses are already reported in the PR jobs list, and checked by GitHub's Merge Queue.)
if: (failure() && github.event.pull_request == null) || (cancelled() && github.event.pull_request == null)
Expand Down
Loading
Loading