Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
79752af
GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow …
raulcd Sep 17, 2025
b715b1f
Add Release steps except upload
raulcd Sep 19, 2025
01b8427
Move back report-ci to run only on scheduled and send report for apac…
raulcd Sep 19, 2025
696f63a
Temporarily add all the almalinux, amazon linux and centos versions t…
raulcd Sep 19, 2025
66df8cb
Remove Centos 7 arm64, remove check for files requiring amd64 (as the…
raulcd Sep 19, 2025
0530d0b
Apply fixes for arm64
raulcd Sep 19, 2025
5f516f3
Leave only almalinux amd64 and arm64
raulcd Sep 19, 2025
5ab9aed
Use correct folder for arm64 prepare
raulcd Sep 19, 2025
1659aaa
Fix target for aarch64
raulcd Sep 19, 2025
45692ce
Several review comments and refactors
raulcd Sep 19, 2025
82c71e3
Add check-labels testing
raulcd Sep 19, 2025
d2f5eb5
Fix labels output
raulcd Sep 19, 2025
58a6dc9
Add some debugging to generate tarball
raulcd Sep 19, 2025
03dc33d
Fix matrix variable to use
raulcd Sep 19, 2025
afa1407
Change matrix.id for {{ matrix.target }}-{{ matrix.architecture }} fo…
raulcd Sep 19, 2025
dab9e38
untar on artifacts-downloaded/{{ matrix.target }}-{{ matrix.architect…
raulcd Sep 22, 2025
78e0ff2
Reduce matrix definition by using id + set up environment variables
raulcd Sep 22, 2025
9a7b054
Add some debian based packages
raulcd Sep 22, 2025
98e9512
Fix architexture to use env variable
raulcd Sep 22, 2025
dbcde95
Define YUM_TARGETS and APT_TARGETS after appending architecture if ne…
raulcd Sep 22, 2025
b336d34
Test all linux-packages together
raulcd Sep 22, 2025
bc8df4f
Only download the specific artifact instead of downloading all of them
raulcd Sep 22, 2025
ec040bb
How does the artifact change when using a specific name?
raulcd Sep 22, 2025
b1b98c8
Untar on dowload script
raulcd Sep 23, 2025
51d7766
Simplify apt:rc:copy and yum:rc:copy to maintain repo folder structur…
raulcd Sep 23, 2025
9528414
Remove old Linux Package tasks
raulcd Sep 23, 2025
7674972
Uncomment the rest of linux package tasks
raulcd Sep 23, 2025
f30f25e
Remove missing tasks
raulcd Sep 23, 2025
471e110
Simplify .tar.gz folder structure more to be able to simplify even mo…
raulcd Sep 23, 2025
516e0fb
Fix verify-yum.sh and verify-apt.sh to remove apt/repositories and yu…
raulcd Sep 23, 2025
c9c4631
Try to fix after understanding the code a bit better :)
raulcd Sep 23, 2025
f4130ef
Revert "Try to fix after understanding the code a bit better :)"
raulcd Sep 25, 2025
b564c22
Revert "Fix verify-yum.sh and verify-apt.sh to remove apt/repositorie…
raulcd Sep 25, 2025
f60d6ad
Revert "Simplify .tar.gz folder structure more to be able to simplify…
raulcd Sep 25, 2025
220dca1
Some fixes to ruby as per code review
raulcd Sep 25, 2025
c7f2c59
Apply suggestion from @kou
raulcd Sep 25, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 75 additions & 0 deletions .github/workflows/check_labels.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

name: Check Labels Reusable

on:
workflow_call:
inputs:
parent-workflow:
description: "The parent workflow filename (without .yml)"
required: true
type: string
outputs:
ci-extra-labels:
description: "The extra CI labels"
value: ${{ jobs.check-labels.outputs.ci-extra-labels }}
force:
description: "Whether to force running the jobs"
value: ${{ jobs.check-labels.outputs.force }}

jobs:
check-labels:
name: Check labels
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
ci-extra-labels: ${{ steps.check.outputs.ci-extra-labels }}
force: ${{ steps.check.outputs.force }}
steps:
- name: Checkout Arrow
if: github.event_name == 'pull_request'
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- name: Check
id: check
env:
GH_TOKEN: ${{ github.token }}
run: |
set -ex
case "${GITHUB_EVENT_NAME}" in
push|schedule|workflow_dispatch)
echo "force=true" >> "${GITHUB_OUTPUT}"
;;
pull_request)
{
echo "ci-extra-labels<<LABELS"
gh pr view ${{ github.event.number }} \
--jq '[.labels[].name | select(startswith("CI: Extra"))]' \
--json labels \
--repo ${GITHUB_REPOSITORY}
echo "LABELS"
} >> "${GITHUB_OUTPUT}"
git fetch origin ${GITHUB_BASE_REF}
git diff --stat origin/${GITHUB_BASE_REF}..
if git diff --stat origin/${GITHUB_BASE_REF}.. | \
grep \
--fixed-strings ".github/workflows/${{ inputs.parent-workflow }}.yml" \
--quiet; then
echo "force=true" >> "${GITHUB_OUTPUT}"
fi
;;
esac
113 changes: 18 additions & 95 deletions .github/workflows/cpp_extra.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ on:
- '!dependabot/**'
paths:
- '.dockerignore'
- '.github/workflows/check_labels.yml'
- '.github/workflows/cpp_extra.yml'
- '.github/workflows/report_ci.yml'
- 'ci/conda_env_*'
- 'ci/docker/**'
- 'ci/scripts/ccache_setup.sh'
Expand All @@ -43,7 +45,9 @@ on:
pull_request:
paths:
- '.dockerignore'
- '.github/workflows/check_labels.yml'
- '.github/workflows/cpp_extra.yml'
- '.github/workflows/report_ci.yml'
- 'ci/conda_env_*'
- 'ci/docker/**'
- 'ci/scripts/ccache_setup.sh'
Expand Down Expand Up @@ -75,53 +79,19 @@ permissions:

jobs:
check-labels:
name: Check labels
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
ci-extra: ${{ steps.check.outputs.ci-extra }}
steps:
- name: Checkout Arrow
if: github.event_name == 'pull_request'
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- name: Check
id: check
env:
GH_TOKEN: ${{ github.token }}
run: |
case "${GITHUB_EVENT_NAME}" in
push|schedule)
ci_extra=true
;;
pull_request)
n_ci_extra_labels=$(
gh pr view ${{ github.event.number }} \
--jq '.labels[].name | select(. == "CI: Extra")' \
--json labels \
--repo ${GITHUB_REPOSITORY} | wc -l)
if [ "${n_ci_extra_labels}" -eq 1 ]; then
ci_extra=true
else
git fetch origin ${GITHUB_BASE_REF}
if git diff --stat origin/${GITHUB_BASE_REF}.. | \
grep \
--fixed-strings ".github/workflows/cpp_extra.yml" \
--quiet; then
ci_extra=true
else
ci_extra=false
fi
fi
;;
esac

echo "ci-extra=${ci_extra}" >> "${GITHUB_OUTPUT}"
uses: ./.github/workflows/check_labels.yml
secrets: inherit
with:
parent-workflow: cpp_extra

docker:
needs: check-labels
name: ${{ matrix.title }}
runs-on: ${{ matrix.runs-on }}
if: needs.check-labels.outputs.ci-extra == 'true'
if: >-
needs.check-labels.outputs.force == 'true' ||
contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') ||
contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: C++')
timeout-minutes: 75
strategy:
fail-fast: false
Expand Down Expand Up @@ -198,7 +168,10 @@ jobs:
needs: check-labels
name: JNI macOS
runs-on: macos-14
if: needs.check-labels.outputs.ci-extra == 'true'
if: >-
needs.check-labels.outputs.force == 'true' ||
contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') ||
contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: C++')
timeout-minutes: 45
env:
MACOSX_DEPLOYMENT_TARGET: "14.0"
Expand Down Expand Up @@ -280,58 +253,8 @@ jobs:
../minimal_build.build/arrow-example

report-extra-cpp:
runs-on: ubuntu-latest
needs:
- docker
- jni-macos
# We don't have the job id as part of the context neither the job name.
# The GitHub API exposes numeric id or job name but not the github.job (report-extra-cpp).
# We match github.job to the name so we can pass it via context in order to be ignored on the report.
# The job is still running.
name: ${{ github.job }}
if: github.event_name == 'schedule' && always()
steps:
- name: Checkout Arrow
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
with:
fetch-depth: 0
- name: Setup Python
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
with:
python-version: 3
- name: Setup Archery
run: python3 -m pip install -e dev/archery[crossbow]
- name: Prepare common options
run: |
if [ "${GITHUB_REPOSITORY}" = "apache/arrow" ]; then
echo "COMMON_OPTIONS=--send" >> "${GITHUB_ENV}"
else
echo "COMMON_OPTIONS=--dry-run" >> "${GITHUB_ENV}"
fi
- name: Send email
env:
GH_TOKEN: ${{ github.token }}
SMTP_PASSWORD: ${{ secrets.ARROW_SMTP_PASSWORD }}
run: |
archery ci report-email \
--ignore ${{ github.job }} \
--recipient-email '[email protected]' \
--repository ${{ github.repository }} \
--sender-email '[email protected]' \
--sender-name Arrow \
--smtp-port 587 \
--smtp-server 'commit-email.info' \
--smtp-user arrow \
${COMMON_OPTIONS} \
${{ github.run_id }}
- name: Send chat message
if: always()
env:
GH_TOKEN: ${{ github.token }}
CHAT_WEBHOOK: ${{ secrets.ARROW_ZULIP_WEBHOOK }}
run: |
archery ci report-chat \
--ignore ${{ github.job }} \
--repository ${{ github.repository }} \
${COMMON_OPTIONS} \
${{ github.run_id }}
uses: ./.github/workflows/report_ci.yml
secrets: inherit
Loading
Loading