- AI DIAL workflows
Continuous Integration instrumentation for AI DIAL components.
Contains reusable workflows for AI-DIAL group of repositories under EPAM GitHub organization.
These workflows could be imported to any repository under EPAM GitHub organization as standard .github/workflows files. See examples below (replace @main with specific version tag).
We expect consumer repositories to follow the given branching strategy:
- A
developmentis the branch for all new work - All code changes are merged from feature branches to
developmentvia pull requests - When there's enough changes in
developmentfor a new release, maintainer cutsrelease-X.Ybranch. Release enters stabilization phase, the branch produce numbered RC artifacts (X.Y.0-rc.N) - Once the maintainer decides the release is stable, he manually (
workflow_dispatch) triggersRelease Workflowforrelease-X.Ybranch withpromoteoption set. StableX.Y.0is published - Since that moment,
release-X.Yenters maintenance phase, and any subsequent pushes to that branch produce patches (X.Y.1,X.Y.2, ...) - Fixes to maintenance branches must be backported from
developmentbranch via cherry-picks
More details can be found in Branching Strategy
Changelog is automatically generated based on git commit history (commit messages) and added to git tags and GitHub releases description
| Scenario | Baseline | Scope |
|---|---|---|
First RC (X.Y.0-rc.0) |
Latest global stable tag | Commits since latest stable tag (full history if no stable tag exists) |
Subsequent RC (X.Y.0-rc.N, N>0) |
Previous RC tag | Commits since previous RC tag only |
Stable promotion (X.Y.0) |
Latest global stable tag | Commits on release branch since cut (full history if no stable tag exists) |
Patch version (X.Y.1+) |
Previous patch version tag | Commits since previous patch only |
- Deduplication: the changelog uses two layers to prevent duplicate changelog entries:
- patch-ID comparison (
git log --cherry-pick): catches cherry-picks with identical changes - commit-message matching: catches conflict-resolved or amended cherry-picks
- patch-ID comparison (
[skip ci] Update versioncommits are excluded from the changelog. They are typically generated by the release workflow and don't contain user-facing changesMerge branchcommits are excluded from the changelog- Changelog entries are grouped by type:
Features,Fixes,Other.Breaking changesare highlighted as a separate section at the top
Tip
Workflows allow independent choices of build artifacts: container image, npm package, or both (default). Can be controlled via docker-build-enabled and publish-enabled inputs respectively. Set inputs values to match your repository needs.
Consumer repository must have:
package.jsonfile withformat,lint,test,build, andpublishscripts defined
package.json
{
"name": "@scope/my-package",
"version": "0.0.0",
"scripts": {
"format": "prettier --check .",
"lint": "eslint .",
"test": "jest",
"build": "tsc",
"publish": "npm publish"
}
}Warning
The version value is updated by CI/CD automation - please do not modify it manually. See more details in Branching section
Tip
We require script names only, not specific implementations - you can use any tools you like as long as you provide the required scripts
Tip
If a build:publishable script exists, it takes precedence over build for the release build (useful in monorepos)
pr.yml
name: PR Workflow
on:
pull_request:
branches: [development, release-*]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
run_tests:
uses: epam/ai-dial-ci/.github/workflows/node_pr.yml@main
secrets: inheritrelease.yml
name: Release Workflow
on:
push:
branches: [development, release-*]
workflow_dispatch:
inputs:
promote:
type: boolean
default: false
description: Promote release to stable (for release-* branches only)
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
release:
uses: epam/ai-dial-ci/.github/workflows/node_release.yml@main
with:
promote: ${{ github.event_name == 'workflow_dispatch' && inputs.promote }}
secrets: inheritConsumer repository must have:
build.gradlewithcheck,checkstyleMainandbuildtasks exposed
build.gradle
plugins {
id "java" // exposes `check`, `build` tasks
id "checkstyle" // exposes `checkstyleMain` task
}
group = "org.example"
version = "0.0.0"Warning
The version value is updated by CI/CD automation - please do not modify it manually. See more details in Branching section
pr.yml
name: PR Workflow
on:
pull_request:
branches: [development, release-*]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
run_tests:
uses: epam/ai-dial-ci/.github/workflows/java_pr.yml@main
secrets: inheritrelease.yml
name: Release Workflow
on:
push:
branches: [development, release-*]
workflow_dispatch:
inputs:
promote:
type: boolean
default: false
description: Promote release to stable (for release-* branches only)
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
release:
uses: epam/ai-dial-ci/.github/workflows/java_release.yml@main
with:
promote: ${{ github.event_name == 'workflow_dispatch' && inputs.promote }}
secrets: inheritWithout a dependency graph, GitHub has no visibility into what packages the Java project uses - so it can't warn when one of them has a known vulnerability (CVE).
The dependency graph snapshot is generated automatically as part of PR Workflow and Release Workflow via integration with the GitHub Dependency Graph Gradle Plugin, and submitted via the GitHub Dependency Submission API.
The generated dependency graph includes all of the dependencies, and is used by GitHub to generate Dependabot Alerts for vulnerable dependencies, as well as to populate the Dependency Graph insights view.
Configuration:
- Navigate to "Settings -> Advanced Security" and enable
Dependency graph,Dependabot alerts,Dependabot security updatesfeatures
Additionally, repository maintainers can benefit from getting automated comments to PRs with insights on dependency changes:
Configuration:
- Ensure dependency graph is enabled as described above
- Navigate to "Settings -> Actions -> General" and set "Workflow permissions" to
Read and write permissions - Commit a workflow file to default branch
dependency-review.yml
name: Dependency Review
on:
workflow_run:
workflows: ["PR Workflow"]
types:
- completed
concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.pull_requests[0].number || github.event.workflow_run.head_sha }}
cancel-in-progress: true
permissions:
actions: read # to download dependency graph artifact
contents: write # to submit dependency graph
pull-requests: read # to resolve PR info for fork PRs
jobs:
dependency-review:
if: |
!github.event.repository.private &&
github.event.workflow_run.event == 'pull_request' &&
github.event.workflow_run.conclusion == 'success'
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@f808768d1510423e83855289c910610ca9b43176 # v2.17.0
with:
disable-telemetry: true
disable-sudo-and-containers: true
egress-policy: block
allowed-endpoints: >
api.github.com:443
api.deps.dev:443
api.securityscorecards.dev:443
- name: Get PR
id: get-pr
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
retries: 3
script: |
const wfRun = context.payload.workflow_run;
const pr = wfRun.pull_requests[0];
let number, base_sha, head_sha;
if (pr) {
number = pr.number;
base_sha = pr.base.sha;
head_sha = pr.head.sha;
} else {
// Fork PR: pull_requests[] is empty, resolve via head branch filter
const headLabel = `${wfRun.head_repository.owner.login}:${wfRun.head_branch}`;
const { data: prs } = await github.rest.pulls.list({
owner: context.repo.owner,
repo: context.repo.repo,
state: 'open',
head: headLabel,
});
const matched = prs[0];
if (!matched) throw new Error(`No open PR found for head ${headLabel}`);
number = matched.number;
base_sha = matched.base.sha;
head_sha = wfRun.head_sha;
}
core.info(`is_fork: ${pr ? 'false' : 'true'}`);
core.info(`number: ${number}`);
core.info(`base_sha: ${base_sha}`);
core.info(`head_sha: ${head_sha}`);
core.setOutput('is_fork', pr ? 'false' : 'true');
core.setOutput('number', number);
core.setOutput('base_sha', base_sha);
core.setOutput('head_sha', head_sha);
- name: Download and submit dependency graph
uses: gradle/actions/dependency-submission@f29f5a9d7b09a7c6b29859002d29d24e1674c884 # v5.0.1
with:
dependency-graph: download-and-submit
- id: dependency-review
uses: actions/dependency-review-action@2031cfc080254a8a887f58cffee85186f0e49e48 # v4.9.0
with:
retry-on-snapshot-warnings: true
retry-on-snapshot-warnings-timeout: 600 # let GitHub process both graphs up to 10 minutes
base-ref: ${{ steps.get-pr.outputs.base_sha }}
head-ref: ${{ steps.get-pr.outputs.head_sha }}
warn-only: true # we don't want to fail the workflow, just to report the issues via comment
show-patched-versions: true
- if: ${{ steps.dependency-review.outputs.comment-content != null }}
name: Save dependency review output report
run: |
cat << 'EOF' > openssf-report.html
${{ steps.dependency-review.outputs.comment-content }}
EOF
- if: ${{ steps.dependency-review.outputs.comment-content != null }}
# Use separate action to comment because the original one can't do it without PR context
uses: marocchino/sticky-pull-request-comment@0ea0beb66eb9baf113663a64ec522f60e49231c0 # v3.0.4
with:
number: ${{ steps.get-pr.outputs.number }}
header: dependency-review
hide_and_recreate: true
path: openssf-report.html
GITHUB_TOKEN: ${{ secrets.ACTIONS_BOT_TOKEN }}
- if: failure()
# If the review fails, we still want to "outdate" the comment to avoid stale information
uses: marocchino/sticky-pull-request-comment@0ea0beb66eb9baf113663a64ec522f60e49231c0 # v3.0.4
with:
number: ${{ steps.get-pr.outputs.number }}
header: dependency-review
hide_and_recreate: true
message: "⚠️ Dependency review workflow failed - results may be outdated. [Check logs](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})"
GITHUB_TOKEN: ${{ secrets.ACTIONS_BOT_TOKEN }}Consumer repository must have:
Makefilefile withlint,build,test,publish(only for Python packages) targets definedpyproject.tomlfile withnameandversiondefined
Makefile
PORT ?= 5001
.PHONY: install lint build test publish
install:
poetry install --all-extras
lint: install
poetry run ruff check .
poetry run ruff format --check .
build: install
poetry build
test: install
if [ -n "$(PYTHON)" ]; then poetry env use "$(PYTHON)"; fi
poetry run pytest
publish: # Required only for Python packages
poetry publish --username __token__ --password $(PYPI_TOKEN) --skip-existingpyproject.toml
[project]
name = "my-package"
version = "0.0.0"Warning
The version value is updated by CI/CD automation - please do not modify it manually. See more details in Branching section
Note
publish target is required only for repositories that produce Python packages as build artifacts
Tip
test target receives Python version, e.g. make test PYTHON=<version>, where <version> is the one defined in code-checks-python-versions workflow input. If multiple versions are defined, the workflow will run tests for each of them in parallel
pr.yml
name: PR Workflow
on:
pull_request:
branches: [development, release-*]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
run_tests:
uses: epam/ai-dial-ci/.github/workflows/python_docker_pr.yml@main
secrets: inheritrelease.yml
name: Release Workflow
on:
push:
branches: [development, release-*]
workflow_dispatch:
inputs:
promote:
type: boolean
default: false
description: Promote release to stable (for release-* branches only)
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
release:
uses: epam/ai-dial-ci/.github/workflows/python_docker_release.yml@main
with:
promote: ${{ github.event_name == 'workflow_dispatch' && inputs.promote }}
secrets: inheritpr.yml
name: PR Workflow
on:
pull_request:
branches: [development, release-*]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
run_tests:
uses: epam/ai-dial-ci/.github/workflows/python_package_pr.yml@main
secrets: inheritrelease.yml
name: Release Workflow
on:
push:
branches: [development, release-*]
workflow_dispatch:
inputs:
promote:
type: boolean
default: false
description: Promote release to stable (for release-* branches only)
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
release:
uses: epam/ai-dial-ci/.github/workflows/python_package_release.yml@main
with:
promote: ${{ github.event_name == 'workflow_dispatch' && inputs.promote }}
secrets: inheritConsumer repository must have:
Makefilewithlinttarget definedDockerfile
Makefile
.PHONY: all lint build run help
all: lint build
build:
docker build -t my-image .
run:
docker run my-image
lint:
docker run --rm -i hadolint/hadolint < Dockerfile
help:
@echo '===================='
@echo 'lint - lint the Dockerfile'
@echo 'build - build docker image'
@echo 'run - run docker image'pr.yml
name: PR Workflow
on:
pull_request:
branches: [development, release-*]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
run_tests:
uses: epam/ai-dial-ci/.github/workflows/generic_docker_pr.yml@main
secrets: inheritrelease.yml
name: Release Workflow
on:
push:
branches: [development, release-*]
workflow_dispatch:
inputs:
promote:
type: boolean
default: false
description: Promote release to stable (for release-* branches only)
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
release:
uses: epam/ai-dial-ci/.github/workflows/generic_docker_release.yml@main
with:
promote: ${{ github.event_name == 'workflow_dispatch' && inputs.promote }}
secrets: inheritpr-title-check.yml
name: "Validate PR title"
on:
pull_request_target:
types:
- opened
- edited
- reopened
permissions: {}
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
pr-title-check:
uses: epam/ai-dial-ci/.github/workflows/pr-title-check.yml@main
secrets:
ACTIONS_BOT_TOKEN: ${{ secrets.ACTIONS_BOT_TOKEN }}slash-command-dispatch.yml
name: Slash Command Dispatch
on:
issue_comment:
types: [created]
jobs:
slashCommandDispatch:
runs-on: ubuntu-latest
if: ${{ github.event.issue.pull_request }}
steps:
- name: Slash Command Dispatch
id: scd
uses: peter-evans/slash-command-dispatch@9bdcd7914ec1b75590b790b844aa3b8eee7c683a # v5.0.2
with:
token: ${{ secrets.ACTIONS_BOT_TOKEN }}
reaction-token: ${{ secrets.ACTIONS_BOT_TOKEN }}
config: >
[
{
"command": "deploy-review",
"permission": "write",
"issue_type": "pull-request",
"repository": "epam/ai-dial-ci",
"static_args": [
"application=${{ github.event.repository.name }}"
]
}
]The E2E testing system for review environments enables automated testing of pull request changes in isolated, short-lived environments. When a developer comments /deploy-review on a PR, the system deploys the PR code to a review environment and runs E2E tests against it.
Flowchart diagram
flowchart-elk TD
A[Developer comments /deploy-review on PR] --> B[Slash Command Dispatch triggers repository_dispatch in epam/ai-dial-ci]
subgraph workflow["deploy-review-command workflow run"]
subgraph deploy["deploy-review job"]
B --> C[deploy-review job]
C --> D{Validate dispatch repository against whitelist}
D -->|Invalid| E[Fail with error]
D -->|Valid| F{Validate required payload values}
F -->|Missing| E
F -->|Present| G[Trigger GitLab deployment pipeline]
end
subgraph e2e["e2e-test job (matrix for each application)"]
G --> I{Does skip-e2e label on PR exist or argument been passed?}
I -->|Yes| J[Skip E2E tests]
I -->|No| K[Wait for application be available]
K --> L[Checkout test repository]
L --> M[Run test action]
end
subgraph monitor["monitor job"]
O[Periodically update PR comment with status]
end
M --> Q[Workflow complete]
J --> Q
end
Q --> R[Final status updated in PR comment]
A test repository must provide composite actions with the following layout:
.github/
actions/
test-<application>/
action.yml <-- Composite action that runs tests
Note
The <application> placeholder must match the application value passed in static_args of the slash-command-dispatch action
The action must have inputs:
environment-url: URL of the deployed review environmentreport-prefix: (Optional) Prefix for report files generated by the actiontest-branch: (Optional) Branch of repository with tests source code to use
Besides inputs, the action will have access to environment variables:
E2E_ADMINE2E_OVERLAY_USERNAMEE2E_PASSWORDE2E_USERNAMENEXT_PUBLIC_OVERLAY_USER_BUCKET
Note
One composite action should test one application only
If you need to disable E2E tests execution:
-
for the whole repository: add
skip-e2eargument tostatic_argslist"static_args": [ "application=${{ github.event.repository.name }}", "skip-e2e" ]
-
for the specific PR: assign
skip-e2elabel to PR -
once: use
/deploy-review skip-e2ecommand in PR comment
cleanup-untagged-images.yml
name: Cleanup untagged images
on:
schedule:
- cron: "0 0 * * *"
jobs:
clean:
name: Delete untagged images
runs-on: ubuntu-latest
permissions:
packages: write
steps:
- uses: dataaxiom/ghcr-cleanup-action@cd0cdb900b5dbf3a6f2cc869f0dbb0b8211f50c4 # v1.0.16
with:
delete-untagged: trueA common case is to trigger development environment(s) update from GitHub to GitLab, e.g. each time a development branch produces a new artifact. Also, it could be not single, but several environments, representing different configuration presets of a single app. To use the example below:
- add a new repository secret with name
DEPLOY_HOSTand value of the gitlab host, e.g.gitlab.example.com - create a new environment, e.g.
development - for the environment, add environment secrets with names
DEPLOY_ACCESS_TOKENandDEPLOY_TRIGGER_TOKENand values of the gitlab access token and trigger token respectively. - use the example workflow file below
deploy-development.yml
name: Deploy development
on:
workflow_dispatch: # manual run
registry_package: # on new package version (main path)
workflow_run: # HACK: redundant trigger to mitigate GitHub's huge delays in registry_package event processing
workflows: ["Release Workflow"]
types:
- completed
jobs:
gitlab-dev-deploy:
if: |
github.event_name == 'workflow_dispatch' ||
github.event.registry_package.package_version.container_metadata.tag.name == 'development' ||
(github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.head_branch == 'development')
uses: epam/ai-dial-ci/.github/workflows/deploy-development.yml@main
with:
gitlab-project-id: "1487"
secrets:
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
DEPLOY_ACCESS_TOKEN: ${{ secrets.DEPLOY_ACCESS_TOKEN }}
DEPLOY_TRIGGER_TOKEN: ${{ secrets.DEPLOY_TRIGGER_TOKEN }}In case of multiple environments, continue creating multiple GitHub environments named after e.g. feature sets, each with its own secrets, then use matrix approach as shown below.
deploy-development.yml
name: Deploy development
on:
workflow_dispatch: # manual run
registry_package: # on new package version (main path)
workflow_run: # HACK: redundant trigger to mitigate GitHub's huge delays in registry_package event processing
workflows: ["Release Workflow"]
types:
- completed
jobs:
trigger:
if: |
github.event_name == 'workflow_dispatch' ||
github.event.registry_package.package_version.container_metadata.tag.name == 'development' ||
(github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.head_branch == 'development')
strategy:
fail-fast: false
matrix:
include:
- environment-name: "development"
gitlab-project-id: "1487"
- environment-name: "feature-1"
gitlab-project-id: "1489"
- environment-name: "feature-2"
gitlab-project-id: "1984"
- environment-name: "feature-3"
gitlab-project-id: "1337"
name: Deploy to ${{ matrix.environment-name }}
uses: epam/ai-dial-ci/.github/workflows/deploy-development.yml@main
with:
gitlab-project-id: ${{ matrix.gitlab-project-id }}
environment-name: ${{ matrix.environment-name }}
secrets:
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
DEPLOY_ACCESS_TOKEN: ${{ secrets.DEPLOY_ACCESS_TOKEN }}
DEPLOY_TRIGGER_TOKEN: ${{ secrets.DEPLOY_TRIGGER_TOKEN }}To change predefined Trivy parameters or set up additional configuration options, create trivy.yaml file in root of your repository. Use example below to add fallback repositories for vulnerabilities and checks DB and thus mitigate rate limit issues.
trivy.yaml
# Trivy configuration file
# https://aquasecurity.github.io/trivy/latest/docs/references/configuration/config-file/
db:
no-progress: true
repository:
- mirror.gcr.io/aquasec/trivy-db:2
- public.ecr.aws/aquasecurity/trivy-db:2
- ghcr.io/aquasecurity/trivy-db:2
java-repository:
- mirror.gcr.io/aquasec/trivy-java-db:1
- public.ecr.aws/aquasecurity/trivy-java-db:1
- ghcr.io/aquasecurity/trivy-java-db:1
misconfiguration:
checks-bundle-repository: mirror.gcr.io/aquasec/trivy-checks:1It's strongly recommended to enable at least Dependabot security updates for your repositories to stay protected from known vulnerabilities in dependencies. To do that, create .github/dependabot.yml file in your repository.
Note
The example for java with gradle as a package manager given. Adjust repo's primary package-ecosystem accordingly. Keep github-actions ecosystem as is
dependabot.yml
version: 2
updates:
- package-ecosystem: "gradle"
directory: "/"
schedule:
interval: "weekly"
day: "wednesday"
time: "09:00"
# Disable version updates, keep security updates only
open-pull-requests-limit: 0
commit-message:
# Prefix all commit messages with "chore: "
prefix: "chore"
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
day: "wednesday"
time: "09:00"
commit-message:
# Prefix all commit messages with "chore: "
prefix: "chore"
groups:
ai-dial-ci:
applies-to: version-updates
patterns:
- "epam/ai-dial-ci/*"
github-actions:
applies-to: version-updates
patterns:
- "*"
exclude-patterns:
- "epam/ai-dial-ci/*"
open-pull-requests-limit: 10Repository maintainers may want to automate approval/merging of Dependabot PRs to reduce manual effort. The example below will automatically approve all Dependabot PRs, and merge those that belong to ai-dial-ci group and are not major version updates.
Warning
The workflow requires auto-merge option in repository settings enabled
Tip
You can modify the conditions to fit your needs, e.g remove steps.metadata.outputs.dependency-group == 'ai-dial-ci' condition to enable auto-merging for all Dependabot PRs except major version updates
dependabot-automation.yml
name: Dependabot Automation
on: pull_request_target
permissions: {}
jobs:
dependabot:
runs-on: ubuntu-latest
if: |
github.event.pull_request.user.login == 'dependabot[bot]' &&
github.repository_owner == 'epam'
steps:
- name: Dependabot metadata
id: metadata
uses: dependabot/fetch-metadata@ffa630c65fa7e0ecfa0625b5ceda64399aea1b36 # v3.0.0
- name: Approve PR
run: gh pr review --approve "$PR_URL"
env:
PR_URL: ${{ github.event.pull_request.html_url }}
GH_TOKEN: ${{ secrets.ACTIONS_BOT_TOKEN }}
- name: Merge PR
if: |
steps.metadata.outputs.dependency-group == 'ai-dial-ci' &&
steps.metadata.outputs.update-type != 'version-update:semver-major'
run: gh pr merge --auto --squash "$PR_URL"
env:
PR_URL: ${{ github.event.pull_request.html_url }}
GH_TOKEN: ${{ secrets.ACTIONS_BOT_TOKEN }}This project contains reusable workflows under .github/workflows directory, and composite actions under actions directory.
Check contribution guidelines for details.