CI: reduce cache size and auto clean caches #10032

yyforyongyu · 2025-07-04T04:06:19Z

This PR reduces the num of caches created per PR, and decreases individual cache size whenever possible by using the use-build-cache flag. It also adds a new action to auto clean caches older than 12 hours to free up space.

Most of the changes are suggested by Gemini.

No need to use build caches

sqlc-check Job: This job only generates SQL models. It doesn't need a build cache.
rpc-check Job: While this job does some building, it's for specific small packages. It's a good candidate to disable the full build cache to save space, as the performance gain is likely minimal compared to the cache cost.
~~check-commits Job: This job only runs a script to check commit messages. It definitely doesn't need a build cache.~~
~~lint Job: Linters perform static analysis and don't need a full build cache.~~
sample-conf-check Job: This job just checks a configuration file.

Combine small jobs

So a single cache can be shared, this also reduces the overhead cost from starting new machines.

Auto cache cleanup

lnd CI is usually very busy, so we want to clean up old caches whenever possible (when the author to have write perms).

gemini-code-assist · 2025-07-04T04:06:24Z

Note

Gemini is unable to generate a summary for this pull request due to the file types involved not being currently supported.

guggero

Thank you very much for looking into optimizing our CI 🙏

.github/workflows/main.yml

guggero · 2025-07-04T06:40:33Z

.github/workflows/main.yml

+      - name: Check commits
+        if: github.event_name == 'pull_request'
+        # ... steps from check-commits job ...
+        run: scripts/check-each-commit.sh upstream/${{ github.base_ref }}


I'm not sure we should do this in the middle. Perhaps we should run all other steps first, since we are going to rebase the branch in the script.

Also, this would benefit from the build cache, as it's explicitly going to compile each commit. So this will take quite a bit longer I assume. But at least it should then not run out of space anymore...

my understanding is that the check commits will rebase from the beginning, and when it finishes we should end up in the same last commit.

Also, this would benefit from the build cache, as it's explicitly going to compile each commit. So this will take quite a bit longer I assume.

You mean by moving it to the end will benefit or?

Yeah, you're right, it should end up with the same diff.
But I think checking each commit is what usually takes a long time, so we should do all "cheap" and quick checks first. Otherwise we spend a lot of CI run time on this to then potentially run into a linter error.

You mean by moving it to the end will benefit or?

No, I mean the check commits step itself would benefit from the build cache (independent of the order). But we've removed the build cache for the whole step.

yeah it does take 22m to run the Static Checks & Lint job, maybe it should be moved into its own job to speed up the feedback? Tho I think the bottom neck is native postgres itest as it takes 57m🤦🏻

hmm further checked a bit - it looks like the usage is really case specific. For instance, this PR has 6 commits, but check commits took 13m, while this PR only takes less than 6m. And since we use make unit it should not use build cache but only package cache? anyway pushing a diff version where we use the build cache to see the difference now.

hmm the check commits now takes longer to finish, roughly 8m with the build cache, maybe i don't understand it at all...the new Static Checks & Lint is faster than the old Static Checks & Lint tho (roughly 2m), the cache size, however, is 2.3 GB vs 440 MB.

ok just realized this can also use the same cache with key prefix unit-test

.github/workflows/main.yml

.github/workflows/stats.yml

yyforyongyu · 2025-07-09T17:58:11Z

/gemini review

Copilot

Pull Request Overview

This PR reduces unnecessary build caches, consolidates similar CI jobs, and adds an automatic cleanup action to remove caches older than 12 hours.

Disable or shrink build caches for jobs that don’t benefit from them
Merge multiple small jobs into a single static-checks job
Introduce an auto-cleanup-cache job and a reusable cleanup-space action

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
.github/workflows/stats.yml	Added top-level permissions and restricted the job to base-repo PRs
.github/workflows/main.yml	Disabled build caches, combined jobs, renamed steps, and added `auto-cleanup-cache` job
.github/actions/cleanup-space/action.yml	New composite action to free runner disk space

Comments suppressed due to low confidence (2)

.github/workflows/stats.yml:9

This workflow posts stats as PR comments but no pull-requests: write permission is granted. Add pull-requests: write under permissions so it can create comments.

  actions: write

.github/workflows/main.yml:528

Rather than matching a comma-separated string, use contains(['OWNER','MEMBER','COLLABORATOR'], github.event.pull_request.author_association) for clearer membership testing.

      contains('OWNER, MEMBER, COLLABORATOR', github.event.pull_request.author_association)

.github/workflows/main.yml

gemini-code-assist

Code Review

This pull request introduces a new GitHub action to clean up runner disk space by removing large, non-essential toolsets. I've suggested adding comments to explain the purpose of removing specific directories for better maintainability.

.github/actions/cleanup-space/action.yml

yyforyongyu · 2025-07-09T18:30:50Z

/gemini review

gemini-code-assist

Code Review

This PR introduces a new action to clean up runner disk space by removing large, non-essential toolsets. The action removes dotnet, android, and ghc to free up space. I've suggested adding comments to explain why these directories are being removed and the approximate disk space freed by each removal.

.github/actions/cleanup-space/action.yml

ellemouton

very nice!

A couple of questions mostly just for my understanding

ellemouton · 2025-07-16T08:19:03Z

.github/workflows/main.yml

@@ -80,6 +81,7 @@ jobs:
        uses: ./.github/actions/setup-go
        with:
          go-version: '${{ env.GO_VERSION }}'
+          use-build-cache: 'no'


would be cool to have a comment either here or in the commit message explaining why it's ok to turn it off here even though we are doing some building. just for future readability.

cool this is now removed - think I got it wrong in the first place

ellemouton · 2025-07-16T08:20:08Z

.github/workflows/main.yml

@@ -113,6 +115,7 @@ jobs:
        uses: ./.github/actions/setup-go
        with:
          go-version: '${{ env.GO_VERSION }}'
+          use-build-cache: 'no'


this one isnt super clear to me. The PR description says

check-commits Job: This job only runs a script to check commit messages. It definitely doesn't need a build cache.

but that is incorrect - it compiles each commit.

Like previous comment, either a comment here or in commit message would help 🙏

yeah right - now removed, also added a commit to explain in details about the caching flow

ellemouton · 2025-07-16T08:21:48Z

.github/workflows/main.yml

@@ -139,6 +142,7 @@ jobs:
        uses: ./.github/actions/setup-go
        with:
          go-version: '${{ env.GO_VERSION }}'
+          use-build-cache: 'no'


also not super clear to me: i think the linter does also analyse compiled code in addition to just pure static checks.

.github/workflows/main.yml

ellemouton · 2025-07-16T08:38:52Z

.github/workflows/main.yml

@@ -537,6 +537,46 @@ jobs:
      - name: 🛡️ backwards compatibility test
        run: make backwards-compat-test

+  #########################################


nice -

do you know why GH doesnt already do this eviction for us? From searching around, it seems like it is meant to auto-evict once we reach 10GB. (1)

yeah i guess it's due to our high frequency that the eviction flow cannot keep up,

The cache eviction process may cause cache thrashing, where caches are created and deleted at a high frequency. To reduce this, you can review the caches for a repository and take corrective steps, such as removing caching from specific workflows.

.github/workflows/stats.yml

This follows GitHub's official recommendation to avoid re-downloading essential tools that setup-go might depend on

Thus saving us from the overhead cost, also they can share the same cache file.

To unify the format used in job names.

This commit adds a more descriptive docs about the caching process. In addition, the `github.job` is now removed from the cache key construction to allow jobs sharing the same cache. In addition, given we now use `inputs.key-prefix`, the `github.job` is no longer relevant as the `key-prefix` already creates a unique name for the cache. We also update the lint and check commits job to use the same cache from the unit test, given they are running in the same environment.

We also make it cancelable.

yyforyongyu self-assigned this Jul 4, 2025

yyforyongyu added CI continuous integration no-changelog labels Jul 4, 2025

guggero reviewed Jul 4, 2025

View reviewed changes

yyforyongyu force-pushed the fix-ci-cache branch 7 times, most recently from 8dd7ad0 to 1d94e0b Compare July 9, 2025 17:54

yyforyongyu requested a review from Copilot July 9, 2025 17:58

Copilot AI reviewed Jul 9, 2025

View reviewed changes

.github/workflows/main.yml Outdated Show resolved Hide resolved

gemini-code-assist bot reviewed Jul 9, 2025

View reviewed changes

.github/actions/cleanup-space/action.yml Outdated Show resolved Hide resolved

yyforyongyu force-pushed the fix-ci-cache branch from 1d94e0b to eba465a Compare July 9, 2025 18:30

gemini-code-assist bot reviewed Jul 9, 2025

View reviewed changes

.github/actions/cleanup-space/action.yml Show resolved Hide resolved

.github/actions/cleanup-space/action.yml Show resolved Hide resolved

.github/actions/cleanup-space/action.yml Show resolved Hide resolved

yyforyongyu requested review from ellemouton and guggero July 15, 2025 17:29

ellemouton approved these changes Jul 16, 2025

View reviewed changes

yyforyongyu added 2 commits July 17, 2025 16:02

.github: use module cache only whenever possible

583b856

.github: replace the aggressive cleanup with a more targeted one

77380de

This follows GitHub's official recommendation to avoid re-downloading essential tools that setup-go might depend on

yyforyongyu force-pushed the fix-ci-cache branch 3 times, most recently from 065eeb9 to 427c1ed Compare July 17, 2025 12:25

yyforyongyu added 3 commits July 17, 2025 20:42

.github: move small tasks onto the same machine

da2b346

Thus saving us from the overhead cost, also they can share the same cache file.

.github: add write permission to delete caches

4f98a03

.github: auto clean caches older than 12 hours

db0927f

yyforyongyu added 2 commits July 17, 2025 20:42

.github: run PR review stats only for side branches

81ce5aa

.github: add new action cleanup-space

4d6f516

yyforyongyu force-pushed the fix-ci-cache branch from 427c1ed to a502e52 Compare July 17, 2025 12:43

yyforyongyu added 3 commits July 17, 2025 20:49

.github: use title case in job names

eb3faf9

To unify the format used in job names.

.github: add timeout when sending coverage reports

d1398f6

We also make it cancelable.

yyforyongyu force-pushed the fix-ci-cache branch from a502e52 to d1398f6 Compare July 17, 2025 13:08

CI: reduce cache size and auto clean caches #10032

Are you sure you want to change the base?

CI: reduce cache size and auto clean caches #10032

Uh oh!

Conversation

yyforyongyu commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

No need to use build caches

Combine small jobs

Auto cache cleanup

Uh oh!

gemini-code-assist bot commented Jul 4, 2025

Uh oh!

guggero left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yyforyongyu Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yyforyongyu commented Jul 9, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

yyforyongyu commented Jul 9, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ellemouton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yyforyongyu commented Jul 4, 2025 •

edited

Loading

yyforyongyu Jul 4, 2025 •

edited

Loading