ci: Use ccache to cache build objects for speeding up building

[bthomee]

High Level Overview of Change

This change enables caching of build objects using ccache on Linux, macOS, and Windows.

Context of Change

Right now, each pipeline invocation builds the source code from scratch. Although compiled Conan dependencies are cached in a remote server, the source build objects are not. We are able to further speed up our builds by leveraging ccache.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Refactor (non-breaking change that only restructures code)
• Performance (increase or change in throughput and/or latency)
• Tests (you added tests for code that already exists, or your new feature included in this PR)
• Documentation update
• Chore (no impact to binary, e.g. .gitignore, formatting, dropping support for older tooling)
• Release

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.1%. Comparing base (41c1be2) to head (40cbe13).

Additional details and impacted files

@@           Coverage Diff           @@
##           develop   #6104   +/-   ##
=======================================
  Coverage     79.1%   79.1%           
=======================================
  Files          836     836           
  Lines        71245   71245           
  Branches      8321    8318    -3     
=======================================
+ Hits         56353   56354    +1     
+ Misses       14892   14891    -1     

see 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mathbunnyru]

The macOS pipelines are executed on bare metal with a ccache directory located outside of the working directory

That's not true that ccache dir is outside of the working directory
ccache can and should be used on macOS (and we do it for Clio)

[mathbunnyru]

Also, please take a look at Clio's implementation; many of the features can be implemented here as well:

• https://github.com/XRPLF/clio/blob/develop/.github/workflows/reusable-build.yml
• https://github.com/XRPLF/clio/blob/develop/cmake/Ccache.cmake (this is better to keep as a separate file, to make our approach more modular, and keep it in sync with Clio)

[bthomee]

The macOS pipelines are executed on bare metal with a ccache directory located outside of the working directory

That's not true that ccache dir is outside of the working directory ccache can and should be used on macOS (and we do it for Clio)

Logging into our macOS GitHub runner shows that the ccache dir is the user's Library/Caches/ccache directory, which is outside of the working directory.

[bthomee]

Also, please take a look at Clio's implementation; many of the features can be implemented here as well:

• https://github.com/XRPLF/clio/blob/develop/.github/workflows/reusable-build.yml
• https://github.com/XRPLF/clio/blob/develop/cmake/Ccache.cmake (this is better to keep as a separate file, to make our approach more modular, and keep it in sync with Clio)

I reluctantly moved the ccache CMake changes back into a separate file, simply for similarity with Clio. However, I don't see in what way this helps keep things more modular, since in my view it just increases the number of files in the directory (albeit by just 1) and it moves 5 lines of compiler-related statements into a different file than where all the compiler-related definitions are stored.

I further looked at the ccache set up in Clio and noticed that upload+download are always enabled together, and generally not enabled for pushes into the develop and release branches or the nightly run; I simplified this by just only enabling ccache for PR commits. This also avoids me having to pass variables between multiple levels of actions.

Note that I use the config_name as the cache key. Since you distinguish between builds with coverage enabled, I also added it to the config name so it becomes a separate entry in the cache.

Also note that sanitizers are not yet enabled, but from the sanitizers PR I recall that they will also be part of the config name. However, @mathbunnyru, I noticed you completely disabled ccache for sanitizers - what's the motivation for doing so?

Finally, what's the motivation for not using ccache for commits into the develop or release branches? It is just to be extra safe or is there an actual known risk with ccache not producing the correct build objects?

[bthomee]

The macOS pipelines are executed on bare metal with a ccache directory located outside of the working directory

That's not true that ccache dir is outside of the working directory ccache can and should be used on macOS (and we do it for Clio)

Logging into our macOS GitHub runner shows that the ccache dir is the user's Library/Caches/ccache directory, which is outside of the working directory.

Alright, I see now that the prepare-runner action overrides this to be inside the working directory.

[bthomee]

@mathbunnyru @vlntb This PR is now ready, and ccache now works on all platforms.

[mathbunnyru]

You always download cache, add files to it (while building new code), and then upload a bigger cache.

This will eventually grow the cache size, and make it useless

[bthomee]

Good point. I had hoped it would be more like Bazel/BuildBuddy where the least recently used files would be dropped automatically once a certain time/size limit was reached, but here the caches are considered as a whole unit and dropped entirely.

My original idea was to use ccache for caching across PRs, so unchanged files wouldn't need to be built again. It seems like that is not possible with the current set up. Should a separate cache be used per PR?

Per actions/cache#1071, one comment states "This functioning is expected and cache is built on the principle that "each key uniquely identifies a cache". if a cache contents have changed and you need to update it, its key should also reflect that.". So would this mean that each commit that modifies some source files needs its own cache? That entirely defeats the purpose...

[mathbunnyru]

Should a separate cache be used per PR?

I don't know if that would work well in rippled's case, because it depends on how active people are pushing their PRs.
I think if there are many active PRs and they override the same cache, it's rarely going to work.

Clio doesn't create caches on PRs, only on commits on develop.
This also solves the problem with fork/non-fork.
And we add a commit hash to the cache id.

[mathbunnyru]

This will all be zeroes and only show ccache size limit, so I don't think it's needed and gives useful information

[bthomee]

It will still show the size of the cache, even though the hits and misses are not shown (because they are zero), e.g.

Local storage:
  Cache size (GB): 0.43 / 5.00 ( 8.64%)

[mathbunnyru]

This information is also shown after the build finishes (with new files included), and on previous upload (without them), that's why it's not that needed

[mathbunnyru]

Does this work when PR is created from fork?

[bthomee]

Yes:

• Everyone has 10GB by default.
• Even if writing fails, the step succeeds. This is apparently a feature (although several developers have filed tickets to complain that they'd like the step to fail the pipeline if saving doesn't work).

Now, if you think it'd be better to not enable this by default in forks, I can make the change. I'm fine either way.

[mathbunnyru]

What will happen if 2 PRs run simultaneously and write to the same cache key?

[bthomee]

My understanding is that the first one to attempt the write will succeed, while the second one will fail to write with the "Unable to reserve cache with key ..., another job may be creating this cache" message.

[mathbunnyru]

Who installed ccache on Windows?
Is it part of deployment scripts?

[bthomee]

It's part of the Terraform script that deploys the Windows runners, using chocolatey to install ccache. Unfortunately it cannot be installed via pip.

[mathbunnyru]

Can't we do it in prepare-runner?
The same way we use brew for Mac, we'll be using cholocatey for Windows

[bthomee]

Done in XRPLF/actions#42.

[mathbunnyru]

You have to update the usage here

[bthomee]

Yes, I will in a follow-up PR.

I'm currently first considering taking advantage of ccache supporting Redis as a remote cache, so I can rip out the largely useless GitHub cache.