ci: reduce cache conflict and usage by erickguan · Pull Request #7642 · apache/opendal

erickguan · 2026-05-30T03:01:03Z

Which issue does this PR close?

None.

Rationale for this change

Reduce cache conflict among jobs.

What changes are included in this PR?

Concurrency group for each binding
Binding test will build cache firstly and reuse them in individual tests.
Slightly reduce cache usage
Various fixes for looking up dependency files
Bump CI action versions

Are there any user-facing changes?

None

AI Usage Statement

No.

tisonkun · 2026-05-30T03:16:39Z


 concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}
+  group: binding_c-${{ github.ref }}-${{ github.event_name }}


Why this change? Isn't it identical?

I want to share the same concurrency group with .github/workflows/test_behavior_binding_c.yml.

Each binding shares the same concurrency group. If there are multiple workflows for a binding, these workflows will run in sequence.

Rollbacked. Tweaking concurrency groups brought more troubles than its worth.

Xuanwo

Thanks a lot for working on this! Only one concern.

Xuanwo · 2026-06-03T11:03:11Z

+          --silent \
+          --show-error \
+          --fail \
+          https://dlcdn.apache.org/hadoop/common/hadoop-3.3.5/hadoop-3.3.5.tar.gz | tar \


We need this cache. ASF's CDN is slow and unstable.

Xuanwo · 2026-06-03T18:19:19Z

6659586

🤯

erickguan · 2026-06-04T04:51:25Z

Hey, I want to test docker and cache approach. Now it's rollback to cache approach.

Test:

docker pull apache/hadoop:3.3.5: ~830MB. Total workflow run ~2m
cache: ~650MB. Total workflow run ~2m

Looks comparable. If we want to save GitHub Action cache quota, it's fine to switch to Docker which is free of charge thanks to Docker policy.

Logs:

docker pull apache/hadoop:3.3.5 logs

Run docker pull apache/hadoop:3.3.5
  docker pull apache/hadoop:3.3.5
  rm -rf /home/runner/hadoop-3.3.5
  # intentionally not remove temp container because
  # GitHub use one-time virtual machine to run a CI run.
  docker cp $(docker create apache/hadoop:3.3.5):/opt/hadoop /home/runner/hadoop-3.3.5
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    RUSTFLAGS: -C force-frame-pointers=yes -C debuginfo=1
    RUST_BACKTRACE: 1
    RUST_LOG: opendal=trace
    CARGO_REGISTRIES_CRATES_IO_PROTOCOL: sparse
    PROTOC_VERSION: 23.4
    PROTOC_ROOT: /home/runner/work/_temp/protoc/23.4
    PROTOC_PLATFORM: linux-x86_64
    ROCKSDB_LIB_DIR: /tmp/rocksdb/lib
    OP_CONNECT_HOST: ***
    OP_CONNECT_TOKEN: ***
    JAVA_HOME: /opt/hostedtoolcache/Java_Zulu_jdk/11.0.31-11/x64
    JAVA_HOME_11_X64: /opt/hostedtoolcache/Java_Zulu_jdk/11.0.31-11/x64
3.3.5: Pulling from apache/hadoop
8ba884070f61: Pulling fs layer
93cf21fbbef2: Pulling fs layer
c778ba593b2f: Pulling fs layer
782cc61249e1: Pulling fs layer
62371aa21ed7: Pulling fs layer
a99f0329f9e1: Pulling fs layer
a3de4f0aebed: Pulling fs layer
126875c5019e: Pulling fs layer
95cfbfe67739: Pulling fs layer
e8adfb95322d: Pulling fs layer
73bf0366d627: Pulling fs layer
08af2d7d7ec2: Pulling fs layer
086c6a12104a: Pulling fs layer
13a0fd9e312b: Pulling fs layer
34a986ee3c28: Pulling fs layer
b08b5eed2a97: Pulling fs layer
36b23ba8b2ef: Pulling fs layer
6503fca807cb: Pulling fs layer
4f4fb700ef54: Pulling fs layer
a40c37f5072c: Pulling fs layer
56b17d50e686: Pulling fs layer
0b5833ca16bc: Pulling fs layer
b08b5eed2a97: Waiting
e8adfb95322d: Waiting
73bf0366d627: Waiting
6503fca807cb: Waiting
4f4fb700ef54: Waiting
08af2d7d7ec2: Waiting
a40c37f5072c: Waiting
086c6a12104a: Waiting
13a0fd9e312b: Waiting
34a986ee3c28: Waiting
a99f0329f9e1: Waiting
782cc61249e1: Waiting
62371aa21ed7: Waiting
a3de4f0aebed: Waiting
56b17d50e686: Waiting
95cfbfe67739: Waiting
126875c5019e: Waiting
0b5833ca16bc: Waiting
36b23ba8b2ef: Waiting
93cf21fbbef2: Verifying Checksum
93cf21fbbef2: Download complete
782cc61249e1: Verifying Checksum
782cc61249e1: Download complete
8ba884070f61: Verifying Checksum
8ba884070f61: Download complete
a99f0329f9e1: Verifying Checksum
a99f0329f9e1: Download complete
62371aa21ed7: Verifying Checksum
62371aa21ed7: Download complete
a3de4f0aebed: Verifying Checksum
a3de4f0aebed: Download complete
126875c5019e: Verifying Checksum
126875c5019e: Download complete
95cfbfe67739: Verifying Checksum
95cfbfe67739: Download complete
e8adfb95322d: Verifying Checksum
e8adfb95322d: Download complete
73bf0366d627: Verifying Checksum
73bf0366d627: Download complete
08af2d7d7ec2: Verifying Checksum
08af2d7d7ec2: Download complete
086c6a12104a: Verifying Checksum
086c6a12104a: Download complete
13a0fd9e312b: Download complete
34a986ee3c28: Verifying Checksum
34a986ee3c28: Download complete
b08b5eed2a97: Verifying Checksum
b08b5eed2a97: Download complete
6503fca807cb: Verifying Checksum
6503fca807cb: Download complete
4f4fb700ef54: Download complete
36b23ba8b2ef: Verifying Checksum
36b23ba8b2ef: Download complete
56b17d50e686: Verifying Checksum
56b17d50e686: Download complete
0b5833ca16bc: Verifying Checksum
0b5833ca16bc: Download complete
c778ba593b2f: Verifying Checksum
c778ba593b2f: Download complete
8ba884070f61: Pull complete
a40c37f5072c: Verifying Checksum
a40c37f5072c: Download complete
93cf21fbbef2: Pull complete
c778ba593b2f: Pull complete
782cc61249e1: Pull complete
62371aa21ed7: Pull complete
a99f0329f9e1: Pull complete
a3de4f0aebed: Pull complete
126875c5019e: Pull complete
95cfbfe67739: Pull complete
e8adfb95322d: Pull complete
73bf0366d627: Pull complete
08af2d7d7ec2: Pull complete
086c6a12104a: Pull complete
13a0fd9e312b: Pull complete
34a986ee3c28: Pull complete
b08b5eed2a97: Pull complete
36b23ba8b2ef: Pull complete
6503fca807cb: Pull complete
4f4fb700ef54: Pull complete
a40c37f5072c: Pull complete
56b17d50e686: Pull complete
0b5833ca16bc: Pull complete
Digest: sha256:2f9b7dcd77ed574eec63a0e570efd4a636d8dd527223b9536bcffe0bb596a921
Status: Downloaded newer image for apache/hadoop:3.3.5
docker.io/apache/hadoop:3.3.5
Run export HADOOP_HOME=/home/runner/hadoop-3.3.5

cache logs

Run actions/cache@v5
  with:
    path: /home/runner/hadoop-3.3.5
    key: cache-hadoop-3.3.5
    enableCrossOsArchive: false
    fail-on-cache-miss: false
    lookup-only: false
    save-always: false
  env:
    RUSTFLAGS: -C force-frame-pointers=yes -C debuginfo=1
    RUST_BACKTRACE: 1
    RUST_LOG: opendal=trace
    CARGO_REGISTRIES_CRATES_IO_PROTOCOL: sparse
    PROTOC_VERSION: 23.4
    PROTOC_ROOT: /home/runner/work/_temp/protoc/23.4
    PROTOC_PLATFORM: linux-x86_64
    ROCKSDB_LIB_DIR: /tmp/rocksdb/lib
    OP_CONNECT_HOST: ***
    OP_CONNECT_TOKEN: ***
    JAVA_HOME: /opt/hostedtoolcache/Java_Zulu_jdk/11.0.31-11/x64
    JAVA_HOME_11_X64: /opt/hostedtoolcache/Java_Zulu_jdk/11.0.31-11/x64
Cache hit for: cache-hadoop-3.3.5
Received 260046848 of 679747974 (38.3%), 247.3 MBs/sec
Received 536870912 of 679747974 (79.0%), 251.1 MBs/sec
Received 679747974 of 679747974 (100.0%), 244.8 MBs/sec
Cache Size: ~648 MB (679747974 B)
/usr/bin/tar -xf /home/runner/work/_temp/7fada215-6f6b-40c2-903a-77c1d9ab1115/cache.tzst -P -C /home/runner/work/opendal/opendal --use-compress-program unzstd
Cache restored successfully
Cache restored from key: cache-hadoop-3.3.5

Xuanwo

Thank you!

erickguan added 2 commits May 30, 2026 10:15

Remove unused cache

d4628da

Reduce concurrent workflows and less conflict in caching

26b77cd

erickguan requested review from Xuanwo and tisonkun as code owners May 30, 2026 03:01

dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. releases-note/ci The PR modifies CI-related content or has a title that begins with "ci" labels May 30, 2026

Various fixes

2aaade1

tisonkun reviewed May 30, 2026

View reviewed changes

erickguan added 3 commits May 30, 2026 11:24

Use lint over check for job names

8d21234

Bump action versions for GitHub CI compatability

d45f3e6

Bump action versions for GitHub CI compatability

0b14ea7

erickguan temporarily deployed to release May 30, 2026 03:39 — with GitHub Actions Inactive

Rollback concurrency changes

e4eea2a

erickguan temporarily deployed to release May 30, 2026 04:05 — with GitHub Actions Inactive

erickguan added 2 commits May 30, 2026 12:09

Improve Java CI cache reuse

ad0b5a7

Fix path

fbe4b54

erickguan temporarily deployed to release May 30, 2026 04:22 — with GitHub Actions Inactive

Reduce cache

ee852fd

erickguan requested a review from tisonkun May 30, 2026 04:39

erickguan temporarily deployed to release May 30, 2026 04:46 — with GitHub Actions Inactive

erickguan changed the title ~~chore(ci): reduce cache conflict and usage~~ ci: reduce cache conflict and usage Jun 2, 2026

Xuanwo reviewed Jun 3, 2026

View reviewed changes

Use docker for hadoop download

6659586

erickguan temporarily deployed to release June 3, 2026 17:42 — with GitHub Actions Inactive

erickguan added 2 commits June 4, 2026 02:50

Fix docker image ref

cb0d8cd

Fix pull image name

58f939c

erickguan temporarily deployed to release June 4, 2026 01:07 — with GitHub Actions Inactive

Rollback to cache download

97715ea

erickguan temporarily deployed to release June 4, 2026 01:31 — with GitHub Actions Inactive

Bump action version

d10ccb6

erickguan temporarily deployed to release June 4, 2026 04:21 — with GitHub Actions Inactive

Add hadoop env back

e25a297

erickguan temporarily deployed to release June 4, 2026 05:00 — with GitHub Actions Inactive

erickguan requested review from Xuanwo June 4, 2026 05:07

Xuanwo approved these changes Jun 5, 2026

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Jun 5, 2026

Xuanwo merged commit 9e6e9cc into main Jun 5, 2026
321 of 323 checks passed

Xuanwo deleted the reduce-cache-usage branch June 5, 2026 08:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: reduce cache conflict and usage#7642

ci: reduce cache conflict and usage#7642
Xuanwo merged 16 commits into
mainfrom
reduce-cache-usage

erickguan commented May 30, 2026 •

edited

Loading

Uh oh!

tisonkun May 30, 2026

Uh oh!

erickguan May 30, 2026

Uh oh!

erickguan May 30, 2026

Uh oh!

Xuanwo left a comment

Uh oh!

Xuanwo Jun 3, 2026

Uh oh!

Xuanwo commented Jun 3, 2026

Uh oh!

erickguan commented Jun 4, 2026 •

edited

Loading

Uh oh!

Xuanwo left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

erickguan commented May 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

AI Usage Statement

Uh oh!

tisonkun May 30, 2026

Choose a reason for hiding this comment

Uh oh!

erickguan May 30, 2026

Choose a reason for hiding this comment

Uh oh!

erickguan May 30, 2026

Choose a reason for hiding this comment

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

Xuanwo Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

Xuanwo commented Jun 3, 2026

Uh oh!

erickguan commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

erickguan commented May 30, 2026 •

edited

Loading

erickguan commented Jun 4, 2026 •

edited

Loading