-
Notifications
You must be signed in to change notification settings - Fork 14.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Simplify caching mechanisms for CI and PROD images
For a long time we had used a sophisticated mechanism to speed up our CI jobs by building the images in "pull_request_target" workflow and pushing them to GitHub registry. That however had several drawbacks: * CI image was complex when it comes to layer setup (we had to pre- cache installed dependencies by installing them from branch tip * The pull_request_target is a very dangerous workflow, we had a number of security problems with it (and it's difficult to debug) * Caching of `pip` and `uv` was not used because it increased size of the image significantly This PR significantly improves the caching mechanisms for the images building of several advacements that were not possible before: * The upload-artifacts@v4 action and improved stash action developed by @assignUser and published in "apache/infrastructure-actions" allows us to store all images (8GB per run) in artifacts rather than in registry - so we can do the image build once and share it with all the jobs. * The uv speed is "enough" to allow occasional installation of Airlfow locally. This allows to utilize cache-mount and locally build uv cache, rather than rely on "remote" cache when we are building local images for breeze. The first time you build local breeze image it will take 2-5 more minutes (depending on your network speed, but because we can utilise cache mounts, every subsequent build should be very fast - even if all dependencies change. Using uv also allows to "always" reinstall airflow when you build the image even if single source file changed, because with cache it takes sub-seconds to reinstall airflow and all dependencies. * the cache mounts are not included in the image size, and since we can export and import images in CI in artifacts and we do not need to rebuild them, the images shared as compressed artifacts are relatively small (2GB) - cache of `uv` is around 4GB on top of that so sharing image built in the "build image" job with other jobs in the same workflow is fast. * we are still using registry cache for the "non-python" parts of the image - both CI and breeze image build speed benefit from using the image cache for system dependencies, database clients etc.
- Loading branch information
Showing
127 changed files
with
2,186 additions
and
2,776 deletions.
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
| # | ||
| --- | ||
| name: 'Prepare all images' | ||
| description: 'Recreates current python image from artifacts' | ||
| inputs: | ||
| pull-image-type: | ||
| description: 'Which image type to prepare' | ||
| default: "CI" | ||
| python-versions-list-as-string: | ||
| description: 'Stringified array of all Python versions to test - separated by spaces.' | ||
| required: true | ||
| platform: | ||
| description: 'Platform for the build - linux/amd64 or linux/arm64' | ||
| required: true | ||
| outputs: | ||
| host-python-version: | ||
| description: Python version used in host | ||
| value: ${{ steps.breeze.outputs.host-python-version }} | ||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - name: "Cleanup docker" | ||
| run: ./scripts/ci/cleanup_docker.sh | ||
| shell: bash | ||
| # TODO: Currently we cannot loop through the list of python versions and have dynamic list of | ||
| # tasks. Instead we hardcode all possible python versions and they - but | ||
| # this should be implemented in stash action as list of keys to download | ||
| - name: "Restore CI docker images ${{ inputs.platform }}-3.8" | ||
| uses: ./.github/actions/prepare_single_image | ||
| with: | ||
| pull-image-type: ${{ inputs.pull-image-type }} | ||
| platform: ${{ inputs.platform }} | ||
| python: "3.8" | ||
| python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }} | ||
| - name: "Restore CI docker images ${{ inputs.platform }}-3.9" | ||
| uses: ./.github/actions/prepare_single_image | ||
| with: | ||
| pull-image-type: ${{ inputs.pull-image-type }} | ||
| platform: ${{ inputs.platform }} | ||
| python: "3.9" | ||
| python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }} | ||
| - name: "Restore CI docker images ${{ inputs.platform }}-3.10" | ||
| uses: ./.github/actions/prepare_single_image | ||
| with: | ||
| pull-image-type: ${{ inputs.pull-image-type }} | ||
| platform: ${{ inputs.platform }} | ||
| python: "3.10" | ||
| python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }} | ||
| - name: "Restore CI docker images ${{ inputs.platform }}-3.11" | ||
| uses: ./.github/actions/prepare_single_image | ||
| with: | ||
| pull-image-type: ${{ inputs.pull-image-type }} | ||
| platform: ${{ inputs.platform }} | ||
| python: "3.11" | ||
| python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }} | ||
| - name: "Restore CI docker images ${{ inputs.platform }}-3.12" | ||
| uses: ./.github/actions/prepare_single_image | ||
| with: | ||
| pull-image-type: ${{ inputs.pull-image-type }} | ||
| platform: ${{ inputs.platform }} | ||
| python: "3.12" | ||
| python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }} | ||
| - name: "Load CI image ${{ inputs.platform }}:${{ inputs.python-versions-list-as-string }}" | ||
| run: | | ||
| for PYTHON in ${{ inputs.python-versions-list-as-string }}; do | ||
| breeze ci-image load --platform ${{ inputs.platform }} --python ${PYTHON} | ||
| rm -rf /tmp/-*${PYTHON}.tar | ||
| done | ||
| shell: bash | ||
| if: inputs.pull-image-type == 'CI' | ||
| - name: "Load PROD image ${{ inputs.platform }}${{ env.PYTHON_MAJOR_MINOR_VERSION }}" | ||
| run: | | ||
| for PYTHON in ${{ inputs.python-versions-list-as-string }}; do | ||
| breeze ci-image load --platform ${{ inputs.platform }} --python ${PYTHON} | ||
| rm -rf /tmp/-*${PYTHON}.tar | ||
| done | ||
| shell: bash | ||
| if: inputs.pull-image-type == 'PROD' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
| # | ||
| --- | ||
| name: 'Prepare single images' | ||
| description: 'Recreates current python image from artifacts' | ||
| inputs: | ||
| python: | ||
| description: 'Python version for image to prepare' | ||
| required: true | ||
| python-versions-list-as-string: | ||
| description: 'Stringified array of all Python versions to prepare - separated by spaces.' | ||
| required: true | ||
| platform: | ||
| description: 'Platform for the build - linux/amd64 or linux/arm64' | ||
| required: true | ||
| outputs: | ||
| host-python-version: | ||
| description: Python version used in host | ||
| value: ${{ steps.breeze.outputs.host-python-version }} | ||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - name: "Restore CI docker images ${{ inputs.platform }}-${{ inputs.python }}" | ||
| uses: apache/infrastructure-actions/stash/restore@c94b890bbedc2fc61466d28e6bd9966bc6c6643c | ||
| with: | ||
| key: "ci-image-save-${{ inputs.platform }}-${{ inputs.python }}" | ||
| path: "/tmp/" | ||
| if: contains(inputs.python-versions-list-as-string, inputs.python) | ||
| - name: "Load CI image ${{ inputs.platform }}${{ env.PYTHON_MAJOR_MINOR_VERSION }}" | ||
| run: breeze ci-image load --platform "${{ inputs.platform }}" --python "${{ inputs.python }}" | ||
| shell: bash | ||
| if: contains(inputs.python-versions-list-as-string, inputs.python) | ||
| - name: "Remove saved image ${{ inputs.platform }}-${{ inputs.python }}" | ||
| run: rm -f /tmp/ci-image-save-*-${{ inputs.python }}* | ||
| shell: bash | ||
| if: contains(inputs.python-versions-list-as-string, inputs.python) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.