CICD update#242
Conversation
- Consolidate CI, pre-commit, and publish workflows into CI.yml and CD.yml - Remove obsolete workflow files - Rename workflow files for clarity - Remove unused SingleRun pipeline code
WalkthroughThis update restructures the project's GitHub Actions workflows, consolidating continuous integration, deployment, and publishing steps. It introduces a new unified CD workflow, modifies the CI workflow to include pre-commit checks, and removes redundant or replaced workflows. Minor adjustments are made to Dockerfile build arguments. Several Dagster asset-related Python modules are deleted. Changes
Sequence Diagram(s)sequenceDiagram
participant GitHub
participant Actions
participant PyPI
participant GHCR
participant Docker
GitHub->>Actions: Push/PR to main or Release
alt Build Job (Python 3.11/3.12)
Actions->>Actions: Checkout, Setup Python, Install deps, Build
end
alt On Release (publish_pypi)
Actions->>Actions: Checkout, Setup Python, Install deps, Build
Actions->>PyPI: Publish package
end
Actions->>Actions: Checkout, Login to GHCR
Actions->>Docker: Build & Push image (Containerfile.cuda)
Actions->>GHCR: Push Docker image
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~18 minutes Poem
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. ✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
.github/workflows/CD.yml (1)
32-51: Optimize PyPI publish job to avoid redundant rebuild.The publish job rebuilds the package unnecessarily since the build job already created artifacts. Consider optimizing by downloading build artifacts instead of rebuilding.
Apply this diff to optimize the publish process:
publish_pypi: if: github.event_name == 'release' needs: build runs-on: ubuntu-latest continue-on-error: true steps: - uses: actions/checkout@v4 + - name: Download build artifacts + uses: actions/download-artifact@v4 + with: + name: dist + path: dist/ - - name: Install uv - uses: astral-sh/setup-uv@v3 - - name: Set up Python 3.12 - run: uv python install 3.12 - - name: Install the project - run: uv sync --all-extras --dev - - name: Run build - run: uv build - name: Publish package uses: pypa/gh-action-pypi-publish@v1.10.3 with: user: __token__ password: ${{ secrets.PYPI_API_TOKEN }}You'll also need to add an artifact upload step to the build job:
- name: Run build run: uv build + - name: Upload build artifacts + uses: actions/upload-artifact@v4 + with: + name: dist + path: dist/
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (9)
.github/workflows/CD.yml(1 hunks).github/workflows/CI.yml(2 hunks).github/workflows/container.yml(0 hunks).github/workflows/pre-commit.yml(0 hunks).github/workflows/python-publish.yml(0 hunks)containers/Containerfile.cpu(1 hunks)containers/Containerfile.cuda(1 hunks)jim_dagster/SingleRun/assets.py(0 hunks)jim_dagster/SingleRun/definitions.py(0 hunks)
💤 Files with no reviewable changes (5)
- jim_dagster/SingleRun/definitions.py
- .github/workflows/python-publish.yml
- .github/workflows/pre-commit.yml
- .github/workflows/container.yml
- jim_dagster/SingleRun/assets.py
🧰 Additional context used
🪛 actionlint (1.7.7)
.github/workflows/CI.yml
43-43: the runner of "actions/checkout@v3" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
45-45: the runner of "actions/setup-python@v3" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
.github/workflows/CD.yml
65-65: the runner of "docker/login-action@v2" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: publish_docker
🔇 Additional comments (6)
containers/Containerfile.cuda (1)
15-15: LGTM! Alignment with Dagster tooling.The change from
--extra pipelineto--extra dagsteraligns the container build with Dagster-specific dependencies, which is consistent with the broader CICD restructuring..github/workflows/CI.yml (2)
1-6: LGTM! Improved workflow specificity.The explicit workflow name and refined triggers (pull requests + main branch pushes only) improve the CI workflow's clarity and scope.
35-35: Minor formatting improvement.Good cleanup of the build step formatting.
containers/Containerfile.cpu (1)
15-15: LGTM! Consistent alignment with Dagster tooling.The change from
--extra pipelineto--extra dagstermaintains consistency with the CUDA container and aligns both containers with Dagster-specific dependencies..github/workflows/CD.yml (2)
3-9: LGTM! Well-structured workflow triggers.The trigger configuration is comprehensive and appropriate:
- Push/PR to main for testing
- Release events for publishing
- Good separation of concerns
12-31: LGTM! Clean build job implementation.The build job properly uses matrix strategy and follows the same pattern as the CI workflow for consistency.
| publish_docker: | ||
| runs-on: ubuntu-latest | ||
| continue-on-error: true | ||
| permissions: | ||
| packages: write | ||
| contents: read | ||
| attestations: write | ||
| id-token: write | ||
| steps: | ||
| - name: Check out the repo | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Log in to GitHub Container Registry | ||
| uses: docker/login-action@v2 | ||
| with: | ||
| registry: ghcr.io | ||
| username: ${{ github.actor }} | ||
| password: ${{ secrets.GITHUB_TOKEN }} | ||
|
|
||
| - name: Extract metadata (tags, labels) for Docker | ||
| id: meta | ||
| uses: docker/metadata-action@v5 | ||
| with: | ||
| images: ghcr.io/${{ github.repository }} | ||
|
|
||
| - name: Build and push Docker image | ||
| id: push | ||
| uses: docker/build-push-action@v5 | ||
| with: | ||
| context: . | ||
| file: ./containers/Containerfile.cuda | ||
| push: true | ||
| tags: ${{ steps.meta.outputs.tags }} | ||
| labels: ${{ steps.meta.outputs.labels }} |
There was a problem hiding this comment.
Update Docker login action version and approve overall structure.
The Docker publish job is well-structured with proper permissions and GHCR integration. However, the static analysis correctly identifies an outdated action version.
Apply this diff to update the outdated action:
- name: Log in to GitHub Container Registry
- uses: docker/login-action@v2
+ uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| publish_docker: | |
| runs-on: ubuntu-latest | |
| continue-on-error: true | |
| permissions: | |
| packages: write | |
| contents: read | |
| attestations: write | |
| id-token: write | |
| steps: | |
| - name: Check out the repo | |
| uses: actions/checkout@v4 | |
| - name: Log in to GitHub Container Registry | |
| uses: docker/login-action@v2 | |
| with: | |
| registry: ghcr.io | |
| username: ${{ github.actor }} | |
| password: ${{ secrets.GITHUB_TOKEN }} | |
| - name: Extract metadata (tags, labels) for Docker | |
| id: meta | |
| uses: docker/metadata-action@v5 | |
| with: | |
| images: ghcr.io/${{ github.repository }} | |
| - name: Build and push Docker image | |
| id: push | |
| uses: docker/build-push-action@v5 | |
| with: | |
| context: . | |
| file: ./containers/Containerfile.cuda | |
| push: true | |
| tags: ${{ steps.meta.outputs.tags }} | |
| labels: ${{ steps.meta.outputs.labels }} | |
| publish_docker: | |
| runs-on: ubuntu-latest | |
| continue-on-error: true | |
| permissions: | |
| packages: write | |
| contents: read | |
| attestations: write | |
| id-token: write | |
| steps: | |
| - name: Check out the repo | |
| uses: actions/checkout@v4 | |
| - name: Log in to GitHub Container Registry | |
| uses: docker/login-action@v3 | |
| with: | |
| registry: ghcr.io | |
| username: ${{ github.actor }} | |
| password: ${{ secrets.GITHUB_TOKEN }} | |
| - name: Extract metadata (tags, labels) for Docker | |
| id: meta | |
| uses: docker/metadata-action@v5 | |
| with: | |
| images: ghcr.io/${{ github.repository }} | |
| - name: Build and push Docker image | |
| id: push | |
| uses: docker/build-push-action@v5 | |
| with: | |
| context: . | |
| file: ./containers/Containerfile.cuda | |
| push: true | |
| tags: ${{ steps.meta.outputs.tags }} | |
| labels: ${{ steps.meta.outputs.labels }} |
🧰 Tools
🪛 actionlint (1.7.7)
65-65: the runner of "docker/login-action@v2" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
🤖 Prompt for AI Agents
In .github/workflows/CD.yml between lines 52 and 85, update the
docker/login-action from version v2 to the latest stable version (e.g., v2.2 or
newer) to ensure compatibility and security. Replace the uses line under "Log in
to GitHub Container Registry" step with the updated version tag while keeping
the rest of the step unchanged. Confirm the overall job structure remains intact
and permissions are correctly set.
| pre-commit: | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| python-version: ["3.11", "3.12"] | ||
| steps: | ||
| - uses: actions/checkout@v3 | ||
| - name: Set up Python ${{ matrix.python-version }} | ||
| uses: actions/setup-python@v3 | ||
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
| - name: Install dependencies | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| python -m pip install pytest | ||
| if [ -f requirements.txt ]; then pip install -r requirements.txt; fi | ||
| python -m pip install . | ||
| - uses: pre-commit/action@v3.0.1 |
There was a problem hiding this comment.
Update action versions and align with build job approach.
The pre-commit job has several issues:
- Outdated actions: Using
actions/checkout@v3andactions/setup-python@v3instead of v4 - Inconsistent setup: Uses
actions/setup-pythonwhile the build job usesuv python install - Manual dependency management: Installs dependencies manually instead of using
uv sync
Apply this diff to align with the build job's approach and update action versions:
- pre-commit:
- runs-on: ubuntu-latest
- strategy:
- fail-fast: false
- matrix:
- python-version: ["3.11", "3.12"]
- steps:
- - uses: actions/checkout@v3
- - name: Set up Python ${{ matrix.python-version }}
- uses: actions/setup-python@v3
- with:
- python-version: ${{ matrix.python-version }}
- - name: Install dependencies
- run: |
- python -m pip install --upgrade pip
- python -m pip install pytest
- if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- python -m pip install .
- - uses: pre-commit/action@v3.0.1
+ pre-commit:
+ runs-on: ubuntu-latest
+ strategy:
+ fail-fast: false
+ matrix:
+ python-version: ["3.11", "3.12"]
+ steps:
+ - uses: actions/checkout@v4
+ - name: Install uv
+ uses: astral-sh/setup-uv@v3
+ - name: Set up Python ${{ matrix.python-version }}
+ run: uv python install ${{ matrix.python-version }}
+ - name: Install the project
+ run: uv sync --all-extras --dev
+ - uses: pre-commit/action@v3.0.1📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| pre-commit: | |
| runs-on: ubuntu-latest | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| python-version: ["3.11", "3.12"] | |
| steps: | |
| - uses: actions/checkout@v3 | |
| - name: Set up Python ${{ matrix.python-version }} | |
| uses: actions/setup-python@v3 | |
| with: | |
| python-version: ${{ matrix.python-version }} | |
| - name: Install dependencies | |
| run: | | |
| python -m pip install --upgrade pip | |
| python -m pip install pytest | |
| if [ -f requirements.txt ]; then pip install -r requirements.txt; fi | |
| python -m pip install . | |
| - uses: pre-commit/action@v3.0.1 | |
| pre-commit: | |
| runs-on: ubuntu-latest | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| python-version: ["3.11", "3.12"] | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Install uv | |
| uses: astral-sh/setup-uv@v3 | |
| - name: Set up Python ${{ matrix.python-version }} | |
| run: uv python install ${{ matrix.python-version }} | |
| - name: Install the project | |
| run: uv sync --all-extras --dev | |
| - uses: pre-commit/action@v3.0.1 |
🧰 Tools
🪛 actionlint (1.7.7)
43-43: the runner of "actions/checkout@v3" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
45-45: the runner of "actions/setup-python@v3" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
🤖 Prompt for AI Agents
In .github/workflows/CI.yml lines 36 to 54, update the pre-commit job to use the
latest action versions by changing actions/checkout@v3 and
actions/setup-python@v3 to v4. Replace the manual Python dependency installation
steps with the uv python install and uv sync commands to align with the build
job's approach, ensuring consistent environment setup and dependency management
across jobs.
1 participant
Summary by CodeRabbit