Ask GitHub for more powerful Action runners

[ChristianKuehnel]

bases on our discussion during the IWG meeting on 2021-08-17:
We should ask GitHub for more powerful runners to get better build performance.

[kwk]

@ChristianKuehnel has someone asked Github for those powerful runners yet?

I've talked with @ldionne last Friday (Nov 26th 2021) and we agreed to write a proposal for the IWG for a way out of this scenario with limited machine power. I assume the resulting proposal shall be an issue here at llvm/llvm-iwg, right? Or do you want it in some other form?

Here's a link to the meeting minutes that you referred to.

[ChristianKuehnel]

has someone asked Github for those powerful runners yet?

Not to my knowledge.

Before we invest in yet another CI system, we should decide on a strategy:

• How many parallel CI systems do we need?
• What builds/checks should be running where?
• What's the central UI for the community?

[lazyparser]

This doc may helps on the estimation:

https://reviews.llvm.org/rG73d52ee7859f

...
The goal
  In 2020, the monorepo had just under 35 thousand commits.  This works
  out to an average of 4 commits per hour.  Already, we can see that a
  builder must cycle in less than 15 minutes to have a hope of being
  useful.  However, those commits are not uniformly distributed.  They
  tend to cluster strongly during US working hours.  Looking at a couple
  of recent (Nov 2021) working days, we routinely see ~10 commits per
  hour during peek times, with occasional spikes as high as ~15 commits
  per hour.  Thus, as a rule of thumb, we should plan for our builder to
  complete ~10-15 builds an hour.
...

[kwk]

has someone asked Github for those powerful runners yet?

Not to my knowledge.

@ChristianKuehnel okay.

Before we invest in yet another CI system, we should decide on a strategy:

* How many parallel CI systems do we need?

Right now I'm just thinking about buildbot and buildkite.

* What builds/checks should be running where?

This implicitly says that we have to decide on the checks before we set up a system. Right now we do have post-merge checks and I can think of a way to enable on-demand pre-merge checks. Then we can see what works and what doesn't or where to improve. @ldionne, in case I'm unclear here I still hope that you understand what I mean ;).

* What's the central UI for the community?

JFY in my proposal the CI is supposed to become a developer's partner and not an invisible and silent opponent as it feels now sometimes. That's why I want to make the user experience as smooth as possible by being able to control bits of the CI through comments. in pull requests. Consider this a dialogue with the CI. It doesn't matter so much what CI sits behind all this but in my case it is buildbot for various reasons. It's certainly the more complicated system compared to buildkite but it has some advantages as well:

1. Buildbot sits nicely between the builder and the developer and we have control over what it does. With buildkite I haven't seen a way author a comment once a build job got accepted. This is needed IMHO in order to let the user know that his request to build a piece of code got accepted.
2. With buildkite we're limited to the platforms for which a buildkite agent exists. While those are numerous, the requirements for buildbot are much less and thereby open up for other platforms.
3. To my best knowledge with buildkite you have pipelines defined as YAML files that sit in your sourcecode. This is great for a fixed and maybe growing set of tests you want to run. But what if some tests are less important in most cases and in others they become super critical? Sure you could control importance by listing files that drive the importance of a test. But I see scenarios in which you want to run a test that is very expensive or very exotic and thus not part of your standard pipeline. Why not let developers ask for those tests to be run on their pull request? That's what I will propose more or less and it will be based on buildbot for now.

I don't want to argue or anything and I still need to do some research so I will just keep my feet still until we have written up the proposal. Just so you know, over the past year I've done a lot of github actions work and learned amazing things that will simplify and streamline my workflow from earlier this year quite a bit.

[tstellar]

We have access to more powerful runners now. See https://reviews.llvm.org/D126506