Skip to content

Commit 200ce2d

Browse files
pietroalbinipietroalbini
authored
Upload JUnit files from Buildomat to Trunk (#9049)
To help triage and handle flaky tests, it was suggested in the control plane huddle a few weeks ago to setup [Trunk's flaky test detection](https://trunk.io/flaky-tests). We run tests on Buildomat though, which doesn't support setting the secret to authenticate with Trunk. To work around the issue, I created a new script to upload all the relevant metadata to Trunk given the GitHub Check Run ID of a Buildomat job: ``` tools/upload_buildomat_junit_to_trunk.sh <github-check-run-id> ``` The script is also hooked up to a GitHub Actions job that runs after every completed Buildomat job. Note that if jobs change names or are added the `case` block in the script will need to be updated.
1 parent 051604d commit 200ce2d

File tree

2 files changed

+178
-0
lines changed

2 files changed

+178
-0
lines changed

‎.github/workflows/upload-buildomat-junit-to-trunk.yml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
name: Upload JUnit from Buildomat to Trunk
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
check_run_id:
7+
description: Buildomat's gong.run.github_id
8+
type: string
9+
required: true
10+
check_run:
11+
types: [completed]
12+
13+
permissions:
14+
contents: read
15+
checks: read
16+
pull-requests: read
17+
18+
jobs:
19+
upload:
20+
name: Upload
21+
runs-on: ubuntu-latest
22+
steps:
23+
- name: Checkout the source code
24+
uses: actions/checkout@v5
25+
26+
- name: Upload JUnit XML to Trunk
27+
run: tools/upload_buildomat_junit_to_trunk.sh $CHECK_RUN_ID
28+
env:
29+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
30+
TRUNK_TOKEN: ${{ secrets.TRUNK_IO_TOKEN }}
31+
TRUNK_ORG_SLUG: oxide
32+
CHECK_RUN_ID: ${{ github.event_name == 'check_run' && github.event.check_run.id || github.event.inputs.check_run_id }}

‎tools/upload_buildomat_junit_to_trunk.sh‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
#!/usr/bin/env bash
2+
# This Source Code Form is subject to the terms of the Mozilla Public
3+
# License, v. 2.0. If a copy of the MPL was not distributed with this
4+
# file, You can obtain one at https://mozilla.org/MPL/2.0/.
5+
6+
# We use Trunk (https://trunk.io) to track flaky tests in our suite. Trunk uses a custom CLI to
7+
# upload JUnit XML files, but unfortunately we cannot run it in Buildomat, as it requires a secret.
8+
#
9+
# To work around that, we use this script to grab the JUnit XML file from a finished Buildomat job
10+
# and upload it. The script requires the GitHub Check Run ID of the Buildomat job, and takes care of
11+
# retrieving the rest of the metadata from the GitHub API.
12+
#
13+
# While the script is meant to be executed in a GitHub Actions workflow triggering after the
14+
# Buildomat job (thanks to the check_run event), it is possible to run the script locally (with a
15+
# valid GITHUB_TOKEN, TRUNK_ORG_SLUG and TRUNK_TOKEN).
16+
17+
set -euo pipefail
18+
IFS=$'\n\t'
19+
20+
if [[ $# -ne 1 ]]; then
21+
echo "usage: $0 <check_run_id>" >&2
22+
exit 1
23+
fi
24+
check_run_id="$1"
25+
26+
tmp="$(mktemp -d)"
27+
trap "rm -rf ${tmp}" EXIT
28+
cd "${tmp}"
29+
30+
mkdir api/
31+
32+
gh api "repos/${GITHUB_REPOSITORY}/check-runs/${check_run_id}" > api/check_run
33+
check_suite_id="$(jq -r .check_suite.id api/check_run)"
34+
commit="$(jq -r .head_sha api/check_run)"
35+
github_app="$(jq -r .app.slug api/check_run)"
36+
job_url="$(jq -r .details_url api/check_run)"
37+
38+
# It is possible for a check run to return more than one PR. This is because you cannot actually
39+
# attach PRs to a check run, the API just returns all PRs with the matching head ref/sha. In the end
40+
# we don't care about which one we pick too much, since this is only reported in the Trunk UI. The
41+
# load bearing thing is whether *at least one* PR is present.
42+
#
43+
# This will evaluate to the raw string `null` if no PR is present, instead of an URL.
44+
pr_url="$(jq -r .pull_requests[0].url api/check_run)"
45+
46+
gh api "repos/${GITHUB_REPOSITORY}/commits/${commit}" > api/commit
47+
author_email="$(jq -r .commit.author.email api/commit)"
48+
author_name="$(jq -r .commit.author.name api/commit)"
49+
commit_message="$(jq -r .commit.message api/commit)"
50+
51+
gh api "repos/${GITHUB_REPOSITORY}/check-suites/${check_suite_id}" > api/check_suite
52+
branch="$(jq -r .head_branch api/check_suite)"
53+
54+
# Determine whether this comes from a PR or not. It's load bearing that we detect things correctly,
55+
# since Trunk treats PRs vs main branch commits differently when analyzing flaky tests.
56+
if [[ "${pr_url}" != "null" ]]; then
57+
gh api "${pr_url}" > api/pr
58+
59+
is_pr=true
60+
pr_number="$(jq -r .number api/pr)"
61+
pr_title="$(jq -r .title api/pr)"
62+
elif [[ "${branch}" == "null" ]]; then
63+
# Unfortunately, the GitHub API doesn't seem to properly detect PRs from forks outside the
64+
# organization. In those cases, the response doesn't include the PR information (failing the
65+
# `if` above) nor the branch name (resulting in a branch name of `null`), and we can assume this
66+
# is a PR from an outside fork.
67+
branch="__unknown_pr__"
68+
69+
is_pr=true
70+
pr_number="99999999"
71+
pr_title="Could not determine the PR"
72+
else
73+
is_pr=false
74+
fi
75+
76+
# Figure out where the JUnit XML report lives and the Trunk variant based on the job name. If new
77+
# jobs are added or renamed, you will need to change this too.
78+
#
79+
# The Trunk variant allows to differentiate flaky tests by platform, and should be the OS name.
80+
job_name="$(jq -r .name api/check_run)"
81+
case "${github_app} / ${job_name}" in
82+
"buildomat / build-and-test (helios)")
83+
artifact_series="junit-helios"
84+
artifact_name="junit.xml"
85+
variant="helios"
86+
;;
87+
"buildomat / build-and-test (ubuntu-22.04)")
88+
artifact_series="junit-linux"
89+
artifact_name="junit.xml"
90+
variant="ubuntu"
91+
;;
92+
*)
93+
echo "unsupported job name, skipping upload: ${job_name}"
94+
exit 0
95+
;;
96+
esac
97+
98+
# Configure the environment to override Trunk's CI detection (with CUSTOM=true) and to provide all
99+
# the relevant information about the CI run. Otherwise Trunk will either pick up nothing (when
100+
# running the script locally) or it will pick up the details of the GHA workflow running this.
101+
function set_var {
102+
echo "setting $1 to $2"
103+
export "$1=$2"
104+
}
105+
set_var CUSTOM true
106+
set_var JOB_URL "${job_url}"
107+
set_var JOB_NAME "${job_name}"
108+
set_var AUTHOR_EMAIL "${author_email}"
109+
set_var AUTHOR_NAME "${author_name}"
110+
set_var COMMIT_BRANCH "${branch}"
111+
set_var COMMIT_MESSAGE "${commit_message}"
112+
if [[ "${is_pr}" == true ]]; then
113+
set_var PR_NUMBER "${pr_number}"
114+
set_var PR_TITLE "${pr_title}"
115+
fi
116+
117+
mkdir upload
118+
cd upload
119+
120+
# Trunk detects which commit the JUnit XML is related to by looking at the HEAD of the current git
121+
# repository. We don't care about the contents of it though, so do a shallow partial clone.
122+
echo >&2
123+
echo "===> cloning commit ${commit} of ${GITHUB_REPOSITORY}..." >&2
124+
git clone --filter=tree:0 --depth=1 "https://github.com/${GITHUB_REPOSITORY}" .
125+
git fetch --depth=1 origin "${commit}"
126+
git checkout "${commit}"
127+
128+
echo >&2
129+
echo "===> downloading the trunk CLI..." >&2
130+
curl -fLO --retry 3 https://trunk.io/releases/trunk
131+
sed -i "s%^#!/bin/bash$%#!/usr/bin/env bash%" trunk # NixOS fix
132+
chmod +x trunk
133+
134+
# The URL is configured through a [[publish]] block in the Buildomat job configuration.
135+
echo >&2
136+
echo "===> downloading the JUnit XML report..." >&2
137+
junit_url="https://buildomat.eng.oxide.computer/public/file/${GITHUB_REPOSITORY}/${artifact_series}/${commit}/${artifact_name}"
138+
curl -fL -o junit.xml --retry 3 "${junit_url}"
139+
140+
echo >&2
141+
echo "===> uploading the JUnit XML report to trunk..." >&2
142+
./trunk flakytests upload \
143+
--junit-paths junit.xml \
144+
--variant "${variant}" \
145+
--org-url-slug "${TRUNK_ORG_SLUG}" \
146+
--token "${TRUNK_TOKEN}"

0 commit comments

Comments
 (0)