-
Notifications
You must be signed in to change notification settings - Fork 3.9k
GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository #47600
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
raulcd
wants to merge
36
commits into
apache:main
Choose a base branch
from
raulcd:GH-47582
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+563
−576
Draft
Changes from all commits
Commits
Show all changes
36 commits
Select commit
Hold shift + click to select a range
79752af
GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow …
raulcd b715b1f
Add Release steps except upload
raulcd 01b8427
Move back report-ci to run only on scheduled and send report for apac…
raulcd 696f63a
Temporarily add all the almalinux, amazon linux and centos versions t…
raulcd 66df8cb
Remove Centos 7 arm64, remove check for files requiring amd64 (as the…
raulcd 0530d0b
Apply fixes for arm64
raulcd 5f516f3
Leave only almalinux amd64 and arm64
raulcd 5ab9aed
Use correct folder for arm64 prepare
raulcd 1659aaa
Fix target for aarch64
raulcd 45692ce
Several review comments and refactors
raulcd 82c71e3
Add check-labels testing
raulcd d2f5eb5
Fix labels output
raulcd 58a6dc9
Add some debugging to generate tarball
raulcd 03dc33d
Fix matrix variable to use
raulcd afa1407
Change matrix.id for {{ matrix.target }}-{{ matrix.architecture }} fo…
raulcd dab9e38
untar on artifacts-downloaded/{{ matrix.target }}-{{ matrix.architect…
raulcd 78e0ff2
Reduce matrix definition by using id + set up environment variables
raulcd 9a7b054
Add some debian based packages
raulcd 98e9512
Fix architexture to use env variable
raulcd dbcde95
Define YUM_TARGETS and APT_TARGETS after appending architecture if ne…
raulcd b336d34
Test all linux-packages together
raulcd bc8df4f
Only download the specific artifact instead of downloading all of them
raulcd ec040bb
How does the artifact change when using a specific name?
raulcd b1b98c8
Untar on dowload script
raulcd 51d7766
Simplify apt:rc:copy and yum:rc:copy to maintain repo folder structur…
raulcd 9528414
Remove old Linux Package tasks
raulcd 7674972
Uncomment the rest of linux package tasks
raulcd f30f25e
Remove missing tasks
raulcd 471e110
Simplify .tar.gz folder structure more to be able to simplify even mo…
raulcd 516e0fb
Fix verify-yum.sh and verify-apt.sh to remove apt/repositories and yu…
raulcd c9c4631
Try to fix after understanding the code a bit better :)
raulcd f4130ef
Revert "Try to fix after understanding the code a bit better :)"
raulcd b564c22
Revert "Fix verify-yum.sh and verify-apt.sh to remove apt/repositorie…
raulcd f60d6ad
Revert "Simplify .tar.gz folder structure more to be able to simplify…
raulcd 220dca1
Some fixes to ruby as per code review
raulcd c7f2c59
Apply suggestion from @kou
raulcd File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,75 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one | ||
# or more contributor license agreements. See the NOTICE file | ||
# distributed with this work for additional information | ||
# regarding copyright ownership. The ASF licenses this file | ||
# to you under the Apache License, Version 2.0 (the | ||
# "License"); you may not use this file except in compliance | ||
# with the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, | ||
# software distributed under the License is distributed on an | ||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
# KIND, either express or implied. See the License for the | ||
# specific language governing permissions and limitations | ||
# under the License. | ||
|
||
name: Check Labels Reusable | ||
|
||
on: | ||
workflow_call: | ||
inputs: | ||
parent-workflow: | ||
description: "The parent workflow filename (without .yml)" | ||
required: true | ||
type: string | ||
outputs: | ||
ci-extra-labels: | ||
description: "The extra CI labels" | ||
value: ${{ jobs.check-labels.outputs.ci-extra-labels }} | ||
force: | ||
description: "Whether to force running the jobs" | ||
value: ${{ jobs.check-labels.outputs.force }} | ||
|
||
jobs: | ||
check-labels: | ||
name: Check labels | ||
runs-on: ubuntu-latest | ||
timeout-minutes: 5 | ||
outputs: | ||
ci-extra-labels: ${{ steps.check.outputs.ci-extra-labels }} | ||
force: ${{ steps.check.outputs.force }} | ||
steps: | ||
- name: Checkout Arrow | ||
if: github.event_name == 'pull_request' | ||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0 | ||
- name: Check | ||
id: check | ||
env: | ||
GH_TOKEN: ${{ github.token }} | ||
run: | | ||
set -ex | ||
case "${GITHUB_EVENT_NAME}" in | ||
push|schedule|workflow_dispatch) | ||
echo "force=true" >> "${GITHUB_OUTPUT}" | ||
;; | ||
pull_request) | ||
{ | ||
echo "ci-extra-labels<<LABELS" | ||
gh pr view ${{ github.event.number }} \ | ||
--jq '[.labels[].name | select(startswith("CI: Extra"))]' \ | ||
--json labels \ | ||
--repo ${GITHUB_REPOSITORY} | ||
echo "LABELS" | ||
} >> "${GITHUB_OUTPUT}" | ||
git fetch origin ${GITHUB_BASE_REF} | ||
git diff --stat origin/${GITHUB_BASE_REF}.. | ||
if git diff --stat origin/${GITHUB_BASE_REF}.. | \ | ||
grep \ | ||
--fixed-strings ".github/workflows/${{ inputs.parent-workflow }}.yml" \ | ||
--quiet; then | ||
echo "force=true" >> "${GITHUB_OUTPUT}" | ||
fi | ||
;; | ||
esac |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -24,7 +24,9 @@ on: | |
- '!dependabot/**' | ||
paths: | ||
- '.dockerignore' | ||
- '.github/workflows/check_labels.yml' | ||
- '.github/workflows/cpp_extra.yml' | ||
- '.github/workflows/report_ci.yml' | ||
- 'ci/conda_env_*' | ||
- 'ci/docker/**' | ||
- 'ci/scripts/ccache_setup.sh' | ||
|
@@ -43,7 +45,9 @@ on: | |
pull_request: | ||
paths: | ||
- '.dockerignore' | ||
- '.github/workflows/check_labels.yml' | ||
- '.github/workflows/cpp_extra.yml' | ||
- '.github/workflows/report_ci.yml' | ||
- 'ci/conda_env_*' | ||
- 'ci/docker/**' | ||
- 'ci/scripts/ccache_setup.sh' | ||
|
@@ -75,53 +79,19 @@ permissions: | |
|
||
jobs: | ||
check-labels: | ||
name: Check labels | ||
runs-on: ubuntu-latest | ||
timeout-minutes: 5 | ||
outputs: | ||
ci-extra: ${{ steps.check.outputs.ci-extra }} | ||
steps: | ||
- name: Checkout Arrow | ||
if: github.event_name == 'pull_request' | ||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0 | ||
- name: Check | ||
id: check | ||
env: | ||
GH_TOKEN: ${{ github.token }} | ||
run: | | ||
case "${GITHUB_EVENT_NAME}" in | ||
push|schedule) | ||
ci_extra=true | ||
;; | ||
pull_request) | ||
n_ci_extra_labels=$( | ||
gh pr view ${{ github.event.number }} \ | ||
--jq '.labels[].name | select(. == "CI: Extra")' \ | ||
--json labels \ | ||
--repo ${GITHUB_REPOSITORY} | wc -l) | ||
if [ "${n_ci_extra_labels}" -eq 1 ]; then | ||
ci_extra=true | ||
else | ||
git fetch origin ${GITHUB_BASE_REF} | ||
if git diff --stat origin/${GITHUB_BASE_REF}.. | \ | ||
grep \ | ||
--fixed-strings ".github/workflows/cpp_extra.yml" \ | ||
--quiet; then | ||
ci_extra=true | ||
else | ||
ci_extra=false | ||
fi | ||
fi | ||
;; | ||
esac | ||
|
||
echo "ci-extra=${ci_extra}" >> "${GITHUB_OUTPUT}" | ||
uses: ./.github/workflows/check_labels.yml | ||
secrets: inherit | ||
with: | ||
parent-workflow: cpp_extra | ||
|
||
docker: | ||
needs: check-labels | ||
name: ${{ matrix.title }} | ||
runs-on: ${{ matrix.runs-on }} | ||
if: needs.check-labels.outputs.ci-extra == 'true' | ||
if: >- | ||
needs.check-labels.outputs.force == 'true' || | ||
contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') || | ||
contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: C++') | ||
timeout-minutes: 75 | ||
strategy: | ||
fail-fast: false | ||
|
@@ -198,7 +168,10 @@ jobs: | |
needs: check-labels | ||
name: JNI macOS | ||
runs-on: macos-14 | ||
if: needs.check-labels.outputs.ci-extra == 'true' | ||
if: >- | ||
needs.check-labels.outputs.force == 'true' || | ||
contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') || | ||
contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: C++') | ||
timeout-minutes: 45 | ||
env: | ||
MACOSX_DEPLOYMENT_TARGET: "14.0" | ||
|
@@ -280,58 +253,8 @@ jobs: | |
../minimal_build.build/arrow-example | ||
|
||
report-extra-cpp: | ||
runs-on: ubuntu-latest | ||
needs: | ||
- docker | ||
- jni-macos | ||
# We don't have the job id as part of the context neither the job name. | ||
# The GitHub API exposes numeric id or job name but not the github.job (report-extra-cpp). | ||
# We match github.job to the name so we can pass it via context in order to be ignored on the report. | ||
# The job is still running. | ||
name: ${{ github.job }} | ||
if: github.event_name == 'schedule' && always() | ||
steps: | ||
- name: Checkout Arrow | ||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0 | ||
with: | ||
fetch-depth: 0 | ||
- name: Setup Python | ||
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0 | ||
with: | ||
python-version: 3 | ||
- name: Setup Archery | ||
run: python3 -m pip install -e dev/archery[crossbow] | ||
- name: Prepare common options | ||
run: | | ||
if [ "${GITHUB_REPOSITORY}" = "apache/arrow" ]; then | ||
echo "COMMON_OPTIONS=--send" >> "${GITHUB_ENV}" | ||
else | ||
echo "COMMON_OPTIONS=--dry-run" >> "${GITHUB_ENV}" | ||
fi | ||
- name: Send email | ||
env: | ||
GH_TOKEN: ${{ github.token }} | ||
SMTP_PASSWORD: ${{ secrets.ARROW_SMTP_PASSWORD }} | ||
run: | | ||
archery ci report-email \ | ||
--ignore ${{ github.job }} \ | ||
--recipient-email '[email protected]' \ | ||
--repository ${{ github.repository }} \ | ||
--sender-email '[email protected]' \ | ||
--sender-name Arrow \ | ||
--smtp-port 587 \ | ||
--smtp-server 'commit-email.info' \ | ||
--smtp-user arrow \ | ||
${COMMON_OPTIONS} \ | ||
${{ github.run_id }} | ||
- name: Send chat message | ||
if: always() | ||
env: | ||
GH_TOKEN: ${{ github.token }} | ||
CHAT_WEBHOOK: ${{ secrets.ARROW_ZULIP_WEBHOOK }} | ||
run: | | ||
archery ci report-chat \ | ||
--ignore ${{ github.job }} \ | ||
--repository ${{ github.repository }} \ | ||
${COMMON_OPTIONS} \ | ||
${{ github.run_id }} | ||
uses: ./.github/workflows/report_ci.yml | ||
secrets: inherit |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.