Skip to content

GH-46499: [CI][Crossbow][C++] Use apache/arrow for Meson #46501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kou merged 1 commit into from
May 30, 2025
Merged

GH-46499: [CI][Crossbow][C++] Use apache/arrow for Meson #46501

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
163 changes: 163 additions & 0 deletions .github/workflows/cpp_extra.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

name: C++ Extra

on:
push:
branches:
- '**'
- '!dependabot/**'
paths:
- '.dockerignore'
- '.github/workflows/cpp_extra.yml'
- 'ci/conda_env_*'
- 'ci/docker/**'
- 'ci/scripts/cpp_*'
- 'ci/scripts/install_azurite.sh'
- 'ci/scripts/install_gcs_testbench.sh'
- 'ci/scripts/install_minio.sh'
- 'ci/scripts/msys2_*'
- 'ci/scripts/util_*'
- 'cpp/**'
- 'docker-compose.yml'
- 'format/Flight.proto'
- 'testing'
tags:
- '**'
pull_request:
paths:
- '.dockerignore'
- '.github/workflows/cpp_extra.yml'
- 'ci/conda_env_*'
- 'ci/docker/**'
- 'ci/scripts/cpp_*'
- 'ci/scripts/install_azurite.sh'
- 'ci/scripts/install_gcs_testbench.sh'
- 'ci/scripts/install_minio.sh'
- 'ci/scripts/msys2_*'
- 'ci/scripts/util_*'
- 'cpp/**'
- 'docker-compose.yml'
- 'format/Flight.proto'
- 'testing'
types:
- labeled
- opened
- reopened
- synchronize
schedule:
- cron: |
0 0 * * *

concurrency:
group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
cancel-in-progress: true

permissions:
contents: read

jobs:
check-labels:
name: Check labels
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
ci-extra: ${{ steps.check.outputs.ci-extra }}
steps:
- name: Check
id: check
env:
GH_TOKEN: ${{ github.token }}
run: |
case "${GITHUB_EVENT_NAME}" in
push|workflow_dispatch)
ci_extra=true
;;
pull_request)
n_ci_extra_labels=$(
gh pr view ${{ github.event.number }} \
--jq '.labels[].name | select(. == "CI: Extra")' \
--json labels \
--repo ${GITHUB_REPOSITORY} | wc -l)
if [ "${n_ci_extra_labels}" -eq 1 ]; then
ci_extra=true
else
ci_extra=false
fi
;;
esac

echo "ci-extra=${ci_extra}" >> "${GITHUB_OUTPUT}"

docker:
needs: check-labels
name: ${{ matrix.title }}
runs-on: ${{ matrix.runs-on }}
if: needs.check-labels.outputs.ci-extra == 'true'
Comment thread
WillAyd marked this conversation as resolved.
Outdated
timeout-minutes: 75
strategy:
fail-fast: false
matrix:
include:
- image: conda-cpp
run-options: >-
-e ARROW_USE_MESON=ON
runs-on: ubuntu-latest
title: AMD64 Ubuntu Meson
env:
ARCHERY_DEBUG: 1
ARROW_ENABLE_TIMING_TESTS: OFF
# DOCKER_VOLUME_PREFIX: ".docker/"
steps:
- name: Checkout Arrow
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0
submodules: recursive
- name: Cache Docker Volumes
uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: .docker
key: extra-${{ matrix.image }}-${{ hashFiles('cpp/**') }}
restore-keys: extra-${{ matrix.image }}-
- name: Setup Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: 3
- name: Setup Archery
run: python3 -m pip install -e dev/archery[docker]
- name: Execute Docker Build
env:
ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
run: |
# GH-40558: reduce ASLR to avoid ASAN/LSAN crashes
sudo sysctl -w vm.mmap_rnd_bits=28
source ci/scripts/util_enable_core_dumps.sh
archery docker run ${{ matrix.run-options || '' }} ${{ matrix.image }}
- name: Docker Push
if: >-
success() &&
github.event_name == 'push' &&
github.repository == 'apache/arrow' &&
github.ref_name == 'main'
env:
ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
continue-on-error: true
run: archery docker push ${{ matrix.image }}
5 changes: 5 additions & 0 deletions ci/scripts/cpp_build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,11 @@ fi
if [ "${ARROW_USE_MESON:-OFF}" = "ON" ]; then
time meson compile -j ${ARROW_BUILD_PARALLEL}
meson install
# Remove all added files in cpp/subprojects/ because they may have
# unreadable permissions on Docker host.
pushd "${source_dir}"
meson subprojects purge --confirm --include-cache
popd
Comment on lines 290 to 294

@kou kou May 23, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@WillAyd This is a workaround.

@kou kou May 23, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... This isn't a workaround... CI failed with this...

@WillAyd WillAyd May 27, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still required?

@kou kou May 27, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. If we don't have this, cpp/subprojects/ has unreadable files. It causes an actions/cache error.

(I think that Meson should not change the source directory.)

@WillAyd WillAyd May 28, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm not sure I understand - Meson strictly forbids changes to the source directory.

Not a blocker for now though - can be cleaned up later if needed

@kou kou May 28, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a shell script that shows that Meson may change the source directory:

set -eux

rm -rf arrow
git clone --recursive https://github.com/apache/arrow.git
cd arrow
ls -lahR cpp/subprojects > before.txt
meson setup cpp.build cpp > /dev/null
meson compile -C cpp.build > /dev/null
ls -lahR cpp/subprojects > after.txt
diff -u {before,after}.txt

Output:

+ rm -rf arrow
+ git clone --recursive https://github.com/apache/arrow.git
Cloning into 'arrow'...
remote: Enumerating objects: 262194, done.
remote: Counting objects: 100% (79/79), done.
remote: Compressing objects: 100% (72/72), done.
remote: Total 262194 (delta 33), reused 13 (delta 7), pack-reused 262115 (from 3)
Receiving objects: 100% (262194/262194), 205.72 MiB | 22.49 MiB/s, done.
Resolving deltas: 100% (185755/185755), done.
Submodule 'cpp/submodules/parquet-testing' (https://github.com/apache/parquet-testing.git) registered for path 'cpp/submodules/parquet-testing'
Submodule 'testing' (https://github.com/apache/arrow-testing) registered for path 'testing'
Cloning into '/arrow/cpp/submodules/parquet-testing'...
remote: Enumerating objects: 421, done.        
remote: Counting objects: 100% (189/189), done.        
remote: Compressing objects: 100% (118/118), done.        
remote: Total 421 (delta 130), reused 79 (delta 55), pack-reused 232 (from 2)        
Receiving objects: 100% (421/421), 1.16 MiB | 19.74 MiB/s, done.
Resolving deltas: 100% (197/197), done.
Cloning into '/arrow/testing'...
remote: Enumerating objects: 992, done.        
remote: Counting objects: 100% (97/97), done.        
remote: Compressing objects: 100% (50/50), done.        
remote: Total 992 (delta 66), reused 58 (delta 47), pack-reused 895 (from 1)        
Receiving objects: 100% (992/992), 29.82 MiB | 23.14 MiB/s, done.
Resolving deltas: 100% (498/498), done.
Submodule path 'cpp/submodules/parquet-testing': checked out '18d17540097fca7c40be3d42c167e6bfad90763c'
Submodule path 'testing': checked out 'd2a13712303498963395318a4eb42872e66aead7'
+ cd arrow
+ ls -lahR cpp/subprojects
+ meson setup cpp.build cpp
+ meson compile -C cpp.build
+ ls -lahR cpp/subprojects
+ diff -u before.txt after.txt
--- before.txt	2025-05-28 09:17:22.268451568 +0900
+++ after.txt	2025-05-28 09:17:47.159847655 +0900
@@ -1,17 +1,124 @@
 cpp/subprojects:
-total 64K
-drwxr-xr-x  2 root root 4.0K May 28 09:17 .
+total 72K
+drwxr-xr-x  4 root root 4.0K May 28 09:17 .
 drwxr-xr-x 12 root root 4.0K May 28 09:17 ..
 -rw-r--r--  1 root root 4.2K May 28 09:17 README.md
 -rw-r--r--  1 root root 1.1K May 28 09:17 azure.wrap
 -rw-r--r--  1 root root 1.2K May 28 09:17 boost.wrap
 -rw-r--r--  1 root root 1.4K May 28 09:17 bzip2.wrap
+drwxrwxr-x  7 root root 4.0K May 28 09:17 gflags-2.2.2
 -rw-r--r--  1 root root 1.1K May 28 09:17 gflags.wrap
 -rw-r--r--  1 root root 1.5K May 28 09:17 google-benchmark.wrap
 -rw-r--r--  1 root root 1.5K May 28 09:17 google-brotli.wrap
 -rw-r--r--  1 root root 1.3K May 28 09:17 google-snappy.wrap
 -rw-r--r--  1 root root 1.5K May 28 09:17 gtest.wrap
 -rw-r--r--  1 root root 1.4K May 28 09:17 lz4.wrap
+drwxr-xr-x  2 root root 4.0K May 28 09:17 packagecache
 -rw-r--r--  1 root root 1.4K May 28 09:17 rapidjson.wrap
 -rw-r--r--  1 root root 1.4K May 28 09:17 zlib.wrap
 -rw-r--r--  1 root root 1.4K May 28 09:17 zstd.wrap
+
+cpp/subprojects/gflags-2.2.2:
+total 136K
+drwxrwxr-x 7 root root 4.0K May 28 09:17 .
+drwxr-xr-x 4 root root 4.0K May 28 09:17 ..
+-rw-rw-r-- 1 root root  113 Nov 12  2018 .gitattributes
+-rw-rw-r-- 1 root root  407 Nov 12  2018 .gitignore
+-rw-rw-r-- 1 root root   93 Nov 12  2018 .gitmodules
+-rw-r--r-- 1 root root   65 May 28 09:17 .meson-subproject-wrap-hash.txt
+-rw-rw-r-- 1 root root  409 Nov 12  2018 .travis.yml
+-rw-rw-r-- 1 root root   32 Nov 12  2018 AUTHORS.txt
+-rw-rw-r-- 1 root root  513 Nov 12  2018 BUILD
+-rw-rw-r-- 1 root root  29K Nov 12  2018 CMakeLists.txt
+-rw-rw-r-- 1 root root 1.5K Nov 12  2018 COPYING.txt
+-rw-rw-r-- 1 root root  13K Nov 12  2018 ChangeLog.txt
+-rw-rw-r-- 1 root root 4.0K Nov 12  2018 INSTALL.md
+-rw-rw-r-- 1 root root  14K Nov 12  2018 README.md
+-rw-rw-r-- 1 root root  255 Nov 12  2018 WORKSPACE
+-rw-rw-r-- 1 root root 1.5K Nov 12  2018 appveyor.yml
+drwxrwxr-x 2 root root 4.0K Nov 12  2018 bazel
+drwxrwxr-x 2 root root 4.0K Nov 12  2018 cmake
+drwxrwxr-x 2 root root 4.0K Nov 12  2018 doc
+drwxrwxr-x 2 root root 4.0K Nov 12  2018 src
+drwxrwxr-x 4 root root 4.0K Nov 12  2018 test
+
+cpp/subprojects/gflags-2.2.2/bazel:
+total 12K
+drwxrwxr-x 2 root root 4.0K Nov 12  2018 .
+drwxrwxr-x 7 root root 4.0K May 28 09:17 ..
+-rw-rw-r-- 1 root root 3.6K Nov 12  2018 gflags.bzl
+
+cpp/subprojects/gflags-2.2.2/cmake:
+total 48K
+drwxrwxr-x 2 root root 4.0K Nov 12  2018 .
+drwxrwxr-x 7 root root 4.0K May 28 09:17 ..
+-rw-rw-r-- 1 root root  238 Nov 12  2018 README_runtime.txt
+-rw-rw-r-- 1 root root 1.2K Nov 12  2018 cmake_uninstall.cmake.in
+-rw-rw-r-- 1 root root 7.5K Nov 12  2018 config.cmake.in
+-rw-rw-r-- 1 root root 1.8K Nov 12  2018 execute_test.cmake
+-rw-rw-r-- 1 root root 2.1K Nov 12  2018 package.cmake.in
+-rw-rw-r-- 1 root root  362 Nov 12  2018 package.pc.in
+-rw-rw-r-- 1 root root 7.7K Nov 12  2018 utils.cmake
+-rw-rw-r-- 1 root root  724 Nov 12  2018 version.cmake.in
+
+cpp/subprojects/gflags-2.2.2/doc:
+total 8.0K
+drwxrwxr-x 2 root root 4.0K Nov 12  2018 .
+drwxrwxr-x 7 root root 4.0K May 28 09:17 ..
+
+cpp/subprojects/gflags-2.2.2/src:
+total 244K
+drwxrwxr-x 2 root root 4.0K Nov 12  2018 .
+drwxrwxr-x 7 root root 4.0K May 28 09:17 ..
+-rw-rw-r-- 1 root root 1.8K Nov 12  2018 config.h
+-rw-rw-r-- 1 root root 1.4K Nov 12  2018 defines.h.in
+-rw-rw-r-- 1 root root  75K Nov 12  2018 gflags.cc
+-rw-rw-r-- 1 root root  30K Nov 12  2018 gflags.h.in
+-rw-rw-r-- 1 root root  27K Nov 12  2018 gflags_completions.cc
+-rw-rw-r-- 1 root root 5.7K Nov 12  2018 gflags_completions.h.in
+-rwxrwxr-x 1 root root 5.1K Nov 12  2018 gflags_completions.sh
+-rw-rw-r-- 1 root root 5.4K Nov 12  2018 gflags_declare.h.in
+-rw-rw-r-- 1 root root 4.0K Nov 12  2018 gflags_ns.h.in
+-rw-rw-r-- 1 root root  17K Nov 12  2018 gflags_reporting.cc
+-rw-rw-r-- 1 root root  15K Nov 12  2018 mutex.h
+-rw-rw-r-- 1 root root  14K Nov 12  2018 util.h
+-rw-rw-r-- 1 root root 2.8K Nov 12  2018 windows_port.cc
+-rw-rw-r-- 1 root root 5.3K Nov 12  2018 windows_port.h
+
+cpp/subprojects/gflags-2.2.2/test:
+total 116K
+drwxrwxr-x 4 root root 4.0K Nov 12  2018 .
+drwxrwxr-x 7 root root 4.0K May 28 09:17 ..
+-rw-rw-r-- 1 root root  12K Nov 12  2018 CMakeLists.txt
+drwxrwxr-x 2 root root 4.0K Nov 12  2018 config
+-rw-rw-r-- 1 root root    9 Nov 12  2018 flagfile.1
+-rw-rw-r-- 1 root root   25 Nov 12  2018 flagfile.2
+-rw-rw-r-- 1 root root   21 Nov 12  2018 flagfile.3
+-rw-rw-r-- 1 root root 1.5K Nov 12  2018 gflags_build.py.in
+-rwxrwxr-x 1 root root  234 Nov 12  2018 gflags_declare_flags.cc
+-rw-rw-r-- 1 root root  348 Nov 12  2018 gflags_declare_test.cc
+-rwxrwxr-x 1 root root 2.5K Nov 12  2018 gflags_strip_flags_test.cc
+-rw-rw-r-- 1 root root  270 Nov 12  2018 gflags_strip_flags_test.cmake
+-rwxrwxr-x 1 root root  52K Nov 12  2018 gflags_unittest.cc
+-rw-rw-r-- 1 root root   28 Nov 12  2018 gflags_unittest_flagfile
+drwxrwxr-x 2 root root 4.0K Nov 12  2018 nc
+
+cpp/subprojects/gflags-2.2.2/test/config:
+total 16K
+drwxrwxr-x 2 root root 4.0K Nov 12  2018 .
+drwxrwxr-x 4 root root 4.0K Nov 12  2018 ..
+-rw-rw-r-- 1 root root  227 Nov 12  2018 CMakeLists.txt
+-rw-rw-r-- 1 root root  579 Nov 12  2018 main.cc
+
+cpp/subprojects/gflags-2.2.2/test/nc:
+total 16K
+drwxrwxr-x 2 root root 4.0K Nov 12  2018 .
+drwxrwxr-x 4 root root 4.0K Nov 12  2018 ..
+-rw-rw-r-- 1 root root  482 Nov 12  2018 CMakeLists.txt
+-rw-rw-r-- 1 root root 2.5K Nov 12  2018 gflags_nc.cc
+
+cpp/subprojects/packagecache:
+total 108K
+drwxr-xr-x 2 root root 4.0K May 28 09:17 .
+drwxr-xr-x 4 root root 4.0K May 28 09:17 ..
+-rw------- 1 root root  97K May 28 09:17 gflags-2.2.2.tar.gz

There is a diff output.

And this is an unreadable file:

+-rw------- 1 root root  97K May 28 09:17 gflags-2.2.2.tar.gz

@kou kou May 28, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was executed on ubuntu:24.04 Docker image with apt update && apt install -y git gcc ninja-build meson cmake.

@WillAyd WillAyd May 28, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah OK - I was excluding subprojects from my consideration of the source directory, as Meson does unpack third party dependencies to that directory during the configure stage (i.e. before the build directory even exists)

else
: ${CMAKE_BUILD_PARALLEL_LEVEL:=${ARROW_BUILD_PARALLEL}}
export CMAKE_BUILD_PARALLEL_LEVEL
Expand Down
2 changes: 2 additions & 0 deletions ci/scripts/cpp_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,9 @@ fi
if [ "${ARROW_USE_MESON:-OFF}" = "ON" ]; then
ARROW_BUILD_EXAMPLES=OFF # TODO: Remove this
meson test \
--no-rebuild \
Comment thread
WillAyd marked this conversation as resolved.
Outdated
--print-errorlogs \
--suite arrow \
"$@"
else
ctest \
Expand Down
10 changes: 0 additions & 10 deletions dev/tasks/tasks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -650,16 +650,6 @@ tasks:
image: {{ image }}
{% endfor %}

test-conda-cpp-meson:
ci: github
template: docker-tests/github.linux.yml
params:
# ARROW_USE_CCACHE=OFF is for using sccache
flags: >-
-e ARROW_USE_CCACHE=OFF
-e ARROW_USE_MESON=ON
image: conda-cpp

test-conda-cpp-valgrind:
ci: github
template: docker-tests/github.linux.yml
Expand Down
Loading