CI/CD Use Cases/Considerations

[ryanmrichard]

Managing the CI for NWChemEx is a tall order. To afford the maintainers the largest amount of creativity possible I propose we put together a list of use cases/considerations. Any solution which addresses these use cases is fine.

Architecture

• Spans several GitHub orgs
  • CI/CD may need to trigger/monitor these organizations
• Many repos (all on GitHub)
  • CI/CD may need to trigger/monitor these repos
• Many dependencies
  • Most can be built by CMaize, but not all of them
  • Have dependency diamonds (two projects, A and B, depend on the same dependency, C, then another project D, depends on both A and B)
  • Want to pin/follow specific versions.
• Code is mix of C++/Python
  • C++17 assumed (and used)
  • Python 3 (not sure about minor we are targeting)
  • Dependencies vary from repo to repo, but there is a set common to many repos
• Documentation is mix of Sphinx-based reST, markdown, and Doxygen
  • Sphinx supports plugins and the plugins needed by each repo may vary
  • N.B. Sphinx plugins are installed via pip
• Build system is CMake based
  • Usually on bleeding edge, but should also maintain some minimum version support
  • Using Ninja under CMake is recommended, but realize make files are the default (i.e. we may want to test as well)
• GitHub actions/workflow is GitHub's native solution
  • Should be preferred as much as possible
  • Dropping down to bash scripts can happen, but realize GitHub can't see contents
  • Large selection of actions available on GitHub, try to avoid writing our own

PR CI (think merging into a dev branch):

• Test multiple operating systems
  • Prioritize Linux and Mac, but Windows would be great...
• Test multiple versions of languages
• Test multiple compilers
• Test multiple versions of dependencies
  • Dependencies we build may couple differently based on compilers
• Test debug and release builds
  • In practice bugs can exist in one vs. the other
• Test optional configurations
  • List of optional configuration varies from repo to repo
• Check files are licensed
• Check files are formatted correctly
• Check code compiles warning free
• Check code for coverage
• Ensure author's have signed contributor agreement
• Ideally should build in less than 5 minutes
  • Will rely on parallelization of workflows, containers, and cacheing
• Test documentation builds
  • Should also build warning free
• Test unit tests pass
• Subsequent runs of failed CI
• Tests should integrate with branch protection rules to avoid accidental merges
• Special workflows for doc only changes?
• PRs may be dependent (changes needed in multiple repos)

Release Deployment (think merging dev to master):

• Longer acceptance tests happen here
• Ensure release builds do not break downstream stake-holders
• Releases should be tagged according to semantic versioning
• Releases should trigger deploying documentation
• Provide containers of releases
  • Likely only a few build types
  • Ideally small size (no extras)
  • Should be built in release mode and as performant as possible
    • Dependencies may need different configurations
    • Configuration will need different options
• Ideally release packages too (Conan, Spack, PyPI, Linux...)
  • Release of packages should be triggered by new releases of source
• A failed dev to master pipeline should revert dev and reopen the original PR with the new issues
  • Fixing a failed dev branch requires a new PR to dev workflow
• May need to do performance testing outside the GitHub ecosystem
• Need to extract module/property type documentation from code developed with PluginPlay
• PRs may be dependent (changes needed in multiple repos)

CI Maintenance

• .github for CI infrastructure which can be factored out
• individual repos for CI specific to repo
• Want to decouple .github maintenance from maintenance of individual repos
  • Adding a decadency to a single repo shouldn't involve .github
  • May need extra workflows for some repos
• The .github repo couples many orgs together
  • Need a way to ensure .github changes don't break everything
• Developer docs so people can go in and work on CI
  • Explanation of overall solution
  • How tos
    • Add a new dependency globally (in .github) vs locally
    • Add new PR checks
    • Add new dev to master checks
• Auto assign PR reviewers (GitHub has a concept of code ownership)
• Notify reviewers when PRs are r2g (maybe a button or a label authors can click/add?)

[ryanmrichard]

@hjjvandam, @twindus do you recall anything else form our CI meetings I missed?

[ryanmrichard]

@zachcran you've been thinking about CI/CD for the CMakePP org, so I'll tag you too.

[hjjvandam]

This looks like a complete list to me.

[twindus]

One piece that I am not sure is represented here is pulling different branches/tagged version of the dependent software. I think it is in the "Test multiple versions of dependencies", but the sub-bullet seems to imply something different.

For the dev documentation builds, do they get deployed somewhere so that the web versions can be checked? I know that can be done locally, but don't know exactly how that would/should be done in CI.

Another question is related to this line: "A failed dev to master pipeline should revert dev and reopen the original PR with the new issues". I am assuming that this means that a merge into dev would trigger trying to merge to master? Or is the merge to master done "by hand"? I believe it has to be the latter. Here I am thinking of the case where you have multiple repos that need to be changed and there will be PRs in different repos. You have to kick them off in some order. Merging to dev in each repo shouldn't be an issue (assuming everything else is OK). But I don't really know how the merge to master will work. If you do the integration tests when you do the first merge to master in one of the repos, this is probably going to fail due to problems associated with not merging in all of the dev branches at once. I am probably missing something here.

Those were the only things that came to mind.

[ryanmrichard]

One piece that I am not sure is represented here is pulling different branches/tagged version of the dependent software. I think it is in the "Test multiple versions of dependencies", but the sub-bullet seems to imply something different.

I added "Want to pin/follow specific versions." to the architecture part.

For the dev documentation builds, do they get deployed somewhere so that the web versions can be checked? I know that can be done locally, but don't know exactly how that would/should be done in CI.

I think in CI the best you get is ensuring the HTML is generated. If the generated HTML contains an error I'm not sure how you'd find that without requiring a human to manually read the documentation (which they should have done in the PR, although it admittedly may render differently). I have the "Releases should trigger deploying documentation" bullet for deploying, but that refers to the production deploymen.

Another question is related to this line: "A failed dev to master pipeline should revert dev and reopen the original PR with the new issues". I am assuming that this means that a merge into dev would trigger trying to merge to master? Or is the merge to master done "by hand"? I believe it has to be the latter. Here I am thinking of the case where you have multiple repos that need to be changed and there will be PRs in different repos. You have to kick them off in some order. Merging to dev in each repo shouldn't be an issue (assuming everything else is OK). But I don't really know how the merge to master will work. If you do the integration tests when you do the first merge to master in one of the repos, this is probably going to fail due to problems associated with not merging in all of the dev branches at once. I am probably missing something here.

I added "PRs may be dependent (changes needed in multiple repos)" to both the PR to dev and dev to master sections. I think we want automatic merging to master. My expectation is that once we're at 1.0 status it should be rare for a dev to master merge to fail. IIRC GitHub added PR queues (or something like that) for ensuring an order; I haven't looked into it too much. Otherwise we could have a flag to request manual merging or something when order matters.

[yzhang-23]

@ryanmrichard I could be naive, but could you explain a little bit more on the dependency diamond issue? I'm not clear what problems it may cause in a CI process.

[ryanmrichard]

Is an example of a dependency diamond in our stack. Namely SimDE depends on ParallelZone through both PluginPlay and Chemist. CMake handles a lot of the problem for you in that CMake will makes sure that SimDE only sees one version of ParallelZone and that the version SimDE sees is compatible with PluginPlay, Chemist, and SimDE.

I don't know what happens when images enter into the equation, in particular is SimDEgoing to see two copies of ParallelZone (one from each base image)? Maybe it'll "all just work", but that's not clear to me.

[yzhang-23]

Is an example of a dependency diamond in our stack. Namely SimDE depends on ParallelZone through both PluginPlay and Chemist. CMake handles a lot of the problem for you in that CMake will makes sure that SimDE only sees one version of ParallelZone and that the version SimDE sees is compatible with PluginPlay, Chemist, and SimDE.

I don't know what happens when images enter into the equation, in particular is SimDEgoing to see two copies of ParallelZone (one from each base image)? Maybe it'll "all just work", but that's not clear to me.

In my current design, the image to build a repo is built in the following steps:

1. Pull the base building image, e. g., the one for building Parallele, as a start point;
2. Add other packages which are necessary but not present in the base image;
3. Pull the release images of dependent repos and copy everything in their /install directories to the current /install directory.

In your example, the building image of SimDe will pull the release images of PluginPlay and Chemist, which both have a library of ParallelZone under their /install directories. However, since the ParallelZone libraries are identical, I would expect that one just overwrite the other, and thus nothing we need to worry about (to be tested)

[zachcran]

However, since the ParallelZone libraries are identical, I would expect that one just overwrite the other, and thus nothing we need to worry about (to be tested)

I believe CMake handles this by just keeping the first one that is found and using it for all of them. There is also an extra safeguard built into CMaize install files that stops the imported target from being created if it already exists, so hopefully "it just works"(TM).

[ryanmrichard]

Is this what you're doing?

[yzhang-23]

Is this what you're doing?

Yes, that's what I'm doing.