GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository #47600

raulcd · 2025-09-19T09:33:13Z

TBD

Rationale for this change

What changes are included in this PR?

Move check_labels and report_ci jobs to independent reusable workflows.
Update cpp_extra to use those.
Create new linux_packaging.yml workflow replicating work that was done on crossbow. Integrate that workflow with check_labels and report_ci
Update release binary submit and binary download to run workflow and download the artifacts from the job instead of from the crossbow repository.

Are these changes tested?

Some via CI on fork and some manual testing.

Are there any user-facing changes?

No

GitHub Issue: [CI][Crossbow][Packaging] Use apache/arrow for linux-packaging jobs #47582

…arrow repository

github-actions · 2025-09-19T09:33:41Z

⚠️ GitHub issue #47582 has been automatically assigned in GitHub to PR creator.

raulcd

Hi @kou I've started working on this and I think I have a working base that could work. I wanted to discuss it with you before moving more packages to validate we are all ok with the approach.
Let me know your thoughts. I have a couple of questions to discuss too, let me know what you think on those too.

.github/workflows/package_linux.yml

dev/release/04-binary-download.sh

raulcd · 2025-09-19T09:49:22Z

.github/workflows/cpp_extra.yml

+    uses: ./.github/workflows/check_labels.yml
+    secrets: inherit
+    with:
+      label: "CI: Extra"


I'd like to move these labels to either:

CI Extra: cpp

CI Extra: CPP

CI: cpp

CI: CPP

So we can have the others the same like:

CI Extra: linux-packaging

CI Extra: Linux Packaging

CI: linux-packaging

CI: Linux Packaging

What do you think?

I like CI: Extra: C++/CI: Extra: Package: Linux or CI: cpp/CI: package: linux/CI: package: linux: apt.

…he/arrow repository

…o validate CI

…y are present on arm64 too and add folder for specific job artifact on check

kou

Thanks!

.github/workflows/check_labels.yml

kou · 2025-09-19T12:07:36Z

.github/workflows/cpp_extra.yml

+    uses: ./.github/workflows/check_labels.yml
+    secrets: inherit
+    with:
+      label: "CI: Extra"


I like CI: Extra: C++/CI: Extra: Package: Linux or CI: cpp/CI: package: linux/CI: package: linux: apt.

.github/workflows/linux_packaging.yml

.github/workflows/package_linux.yml

.github/workflows/linux_packaging.yml

dev/release/04-binary-download.sh

kou · 2025-09-23T12:15:22Z

dev/release/binary-task.rb

-            # apache/arrow-adbc uses debian-bookworm. So the following
-            # glob must much both of them.


Yes.

But I can change ADBC to simplify this script.

kou · 2025-09-23T12:36:29Z

.github/workflows/package_linux.yml

+        run: |
+          set -ex
+          pushd dev/tasks/linux-packages
+          tar cvzf ${{ matrix.id }}.tar.gz */${TASK_NAMESPACE}/repositories


Let's simplify paths more to simplify binary-task.rb:

Suggested change

tar cvzf ${{ matrix.id }}.tar.gz */${TASK_NAMESPACE}/repositories

mkdir -p artifacts

cp -a */${TASK_NAMESPACE}/repositories/* artifacts/

tar cvzf ${{ matrix.id }}.tar.gz -C artifacts .

I have tried to simplify the paths slightly more but I am finding quite difficult to understand how all tangles together. How do we generate the repositories from those rpms, how we merge these files with other existing files from the repo, how do we create the testing repositories, etcetera so I am reverting my latest changes. I am happy to leave it like this (current generated structure) and try to simplify on a future PR or if you want to guide me on how to do it or if you want to push to this branch.

kou · 2025-09-23T12:37:19Z

dev/release/04-binary-download.sh

+
+# Wait for the GitHub Workflow that creates the Linux packages
+# to finish before downloading the artifacts.
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"


We don't need . here because all information is passed via arguments:

Suggested change

. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"

"${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"

kou · 2025-09-23T12:45:37Z

dev/release/binary-task.rb

-            # glob must much both of them.
-            Dir.glob("#{source_dir_prefix}*/*") do |path|
-              base_name = File.basename(path)
+            destination_prefix = "#{incoming_dir}/#{distribution}"


Thanks for writing Ruby!

Sure. I can help. Can I push something to this branch if it's needed?

BTW, destination_dir is better for this variable name because this is a directory name not a prefix of path. (In this code, "prefix" is used for abc for abcde directory and "dir" is used for abcde directory.)

dev/release/binary-task.rb

kou · 2025-09-23T12:48:34Z

dev/release/post-05-update-gh-release-notes.sh

 # Wait for the GitHub Workflow that creates the GitHub Release
 # to finish before updating the release notes.
-"${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${TAG}" "${WORKFLOW}"
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${TAG}" "${WORKFLOW}"


We don't need . here.

dev/tasks/linux-packages/helper.rb

kou · 2025-09-23T12:57:11Z

dev/release/utils-watch-gh-workflow.sh

+download_artifacts() {
+  RUN_ID=$(get_run_id)
+  echo "Downloading artitfacts for workflow with ID: ${RUN_ID}"
+  gh run download \
+    ${RUN_ID} \
+    --repo "${REPOSITORY}" \
+    --dir "$1"
+}


Hmm. Defining this function in this file isn't good because this file name is watch-gh-workflow.

We could rename the file to utils-gh-workflow.sh but if we use release for the artifacts this won't be necessary.

kou · 2025-09-23T12:57:38Z

dev/release/04-binary-download.sh

+# to finish before downloading the artifacts.
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"
+
+RUN_ID=$(get_run_id)


Ah, you want to use a function defined in utils-watch-gh-workflow.sh.

kou · 2025-09-23T12:58:39Z

dev/release/04-binary-download.sh

+
+RUN_ID=$(get_run_id)
+# Download the artifacts created by the package_linux.yml workflow
+download_artifacts "${SOURCE_DIR}/../../packages/${CROSSBOW_JOB_ID}"


How about using GitHub Release for RC instead of GitHub Actions artifacts? If we use GitHub Release, we don't need RUN_ID: gh release download apache-arrow-${VERSION}-rc${RC} ...

so you mean, for scheduled or PR/push use artifacts but if workflow_dispatch was used, which is trigger of RC, upload to release? I don't want to create a release every time the job is executed but that could work. In that case we might use artifacts to test and upload to release if tests are successful and job type was workflow_dispatch

I want to upload artifacts to GitHub Release only when apache-arrow-${VERSION}-rc${RC} tag is pushed. Can we use on.push.tags instead of workflow_dispatch for RC? (I can provide changes for the approach.)

…re binary tasks

…m/repositories subfolders Try fixing yum tests

This reverts commit c9c4631.

…s and yum/repositories subfolders" This reverts commit 516e0fb.

… even more binary tasks" This reverts commit 471e110.

Co-authored-by: Sutou Kouhei <[email protected]>

raulcd · 2025-09-25T08:47:26Z

@kou I think there are two main things missing here:

whether we want to further simplify the repositories structure and the .tar.gz (this can be done on a separate PR)
whether we want to use the tag instead of workflow dispatch for the release and use gh release download instead of gh run download. If we don't want to do this on this PR I should probably rename utils-watch-gh-workflow.sh to utils-gh-workflow.sh so I can source the file and add the new function.

I can work on the second point regardless of the approach (unless you want to do the tag part).

kou · 2025-09-25T09:39:35Z

Can I work on this next week? I'll push something to this branch.

raulcd · 2025-09-25T10:08:00Z

Can I work on this next week? I'll push something to this branch.

Absolutely. I will try to deeply review the changes for simplifying the repositories structure so I can learn more about how we build those packages. It would be good to spread the knowledge to help maintaining those.
Thank you very much @kou .

raulcd added 2 commits September 19, 2025 10:10

apacheGH-47582: [CI][Packaging] Move linux-packaging tasks to apache/…

79752af

…arrow repository

Add Release steps except upload

b715b1f

github-actions bot added the awaiting committer review Awaiting committer review label Sep 19, 2025

raulcd commented Sep 19, 2025

View reviewed changes

.github/workflows/package_linux.yml Outdated Show resolved Hide resolved

dev/release/04-binary-download.sh Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Sep 19, 2025

raulcd commented Sep 19, 2025

View reviewed changes

Move back report-ci to run only on scheduled and send report for apac…

01b8427

…he/arrow repository

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 19, 2025

Temporarily add all the almalinux, amazon linux and centos versions t…

696f63a

…o validate CI

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Sep 19, 2025

Remove Centos 7 arm64, remove check for files requiring amd64 (as the…

66df8cb

…y are present on arm64 too and add folder for specific job artifact on check

raulcd force-pushed the GH-47582 branch from 9e3e8e4 to 66df8cb Compare September 19, 2025 11:30

raulcd added 3 commits September 19, 2025 14:25

Apply fixes for arm64

0530d0b

Leave only almalinux amd64 and arm64

5f516f3

Use correct folder for arm64 prepare

5ab9aed

kou reviewed Sep 19, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 19, 2025

Fix target for aarch64

1659aaa

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 19, 2025

Several review comments and refactors

45692ce

raulcd force-pushed the GH-47582 branch from 8a2e9d6 to 45692ce Compare September 19, 2025 14:37

Add check-labels testing

82c71e3

github-actions bot added the awaiting changes Awaiting changes label Sep 23, 2025

Remove missing tasks

f30f25e

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 23, 2025

kou reviewed Sep 23, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 23, 2025

Simplify .tar.gz folder structure more to be able to simplify even mo…

471e110

…re binary tasks

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Sep 23, 2025

raulcd force-pushed the GH-47582 branch from 29fe2eb to b9a5de1 Compare September 23, 2025 15:33

Fix verify-yum.sh and verify-apt.sh to remove apt/repositories and yu…

516e0fb

…m/repositories subfolders Try fixing yum tests

raulcd force-pushed the GH-47582 branch from b9a5de1 to 516e0fb Compare September 23, 2025 16:54

raulcd added 4 commits September 23, 2025 21:12

Try to fix after understanding the code a bit better :)

c9c4631

Revert "Try to fix after understanding the code a bit better :)"

f4130ef

This reverts commit c9c4631.

Revert "Fix verify-yum.sh and verify-apt.sh to remove apt/repositorie…

b564c22

…s and yum/repositories subfolders" This reverts commit 516e0fb.

Revert "Simplify .tar.gz folder structure more to be able to simplify…

f60d6ad

… even more binary tasks" This reverts commit 471e110.

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 25, 2025

Some fixes to ruby as per code review

220dca1

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 25, 2025

Apply suggestion from @kou

c7f2c59

Co-authored-by: Sutou Kouhei <[email protected]>

		# apache/arrow-adbc uses debian-bookworm. So the following
		# glob must much both of them.

-          tar cvzf ${{ matrix.id }}.tar.gz */${TASK_NAMESPACE}/repositories
+          mkdir -p artifacts
+          cp -a */${TASK_NAMESPACE}/repositories/* artifacts/
+          tar cvzf ${{ matrix.id }}.tar.gz -C artifacts .

	. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"
	"${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"

GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository #47600

Are you sure you want to change the base?

GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository #47600

Conversation

raulcd commented Sep 19, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TBD

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Sep 19, 2025

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

raulcd Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raulcd commented Sep 25, 2025

Uh oh!

kou commented Sep 25, 2025

Uh oh!

raulcd commented Sep 25, 2025

Uh oh!

Uh oh!

raulcd commented Sep 19, 2025 •

edited by github-actions bot

Loading

raulcd Sep 19, 2025 •

edited

Loading