Move to dev/prod model?

[ryanmrichard]

I'm wondering if we want to adopt a dev/prod model. Basically this means:

• When a PR gets accepted it does not go into master, it goes into a branch dev.
• Once a push to dev happens it triggers a different series of CI tests
  • Usually this is where longer tests would run (only quick tests need to pass to get merged into dev)
  • This is where we could ensure/determine if the changes break downstream repos (we'd thus avoid breaking downstream repos until dev gets merged into master)
  • If these CI tests pass, then we'd merge dev into master.

I'm only opening this issue on ParallelZone, but it would apply throughout the stack.

@NWChemEx-Project/ci-cd thoughts?

[yzhang-23]

I think this is a better model than our current one. Do we have those longer tests ready?

[ryanmrichard]

@yzhang-23 not yet.

[hjjvandam]

I think the idea is a good one. One question here is: what takes longer, building the code or running the tests?
At the moment I think building the code takes longer than running the tests. However, if we want to have tests that check the code more extensively this may become an issue. For example building a Fock matrix once given synthetic data could be quite fast as compared to doing a full direct SCF with real integrals, etc. One thing that will be needed in particular is some guidance on what is long and what is short. Is 1 second still short, what about 10 seconds, ...? 100 Tests of 10 seconds each still takes about 20 minutes.

[ryanmrichard]

@hjjvandam at the moment building the code takes longer. My hope is that between the roll out of Docker images, and cacheing, we can get the build time way down (I actually am optimistic we can get each CI run to under 5 minutes).

I'd be inclined to consider long vs. short more descriptive than prescriptive. Point being, I think merging into dev should focus on getting the most code coverage, in the least amount of time, whereas dev to master should focus on exhaustive coverage. The assumption is that exhaustive coverage is going to require, in some shape or another, significantly more resources than partial coverage. If that assumption is not valid, then I think the separation doesn't make as much sense; however, particularly if we throw some performance tests into the mix, I feel like it's a pretty safe assumption.

[twindus]

I am probably missing something, but what are the major advantages/disadvantages of this approach. I see the major one of not breaking downstream repos in the master, but won't they still break in dev? Won't this still require a lot of management of keeping dev and master up to date with each other? I do see potential advantages in the testing time, so that would be useful.

[yzhang-23]

@yzhang-23 not yet.

If those longer (regression) tests are not ready, I suggest that we should get them ready, then move to the dev/prod model.

[ryanmrichard]

Assume I make a PR which breaks downstream repos. With dev/master, we can safely merge the PR into dev without breaking the downstream repos (they use master). While we work on fixes for the downstream repos (which is facilitated by being able to stage those changes into the downstream dev branches), other developers can develop off of the dev branches and development doesn't have to stop on upstream or downstream repos.

In theory I think the only additional management is creating the additional automation. More specifically, if changes don't break downstream repos they get automerged from dev to master. In theory the dev to master pushes should be fully automated, i.e., if dev doesn't break downstream then it gets immediately merged to master; if it does break the downstream repos it needs to wait for the downstream repos' dev branches to work with the changes.

[ryanmrichard]

If those longer (regression) tests are not ready, I suggest that we should get them ready, then move to the dev/prod model.

The tests will come (simply making sure we don't break downstream repos is going to be time consuming). The bigger question is how hard will it be to roll out a dev/prod model when those tests are ready vs. right now as we make the CI changes we're already in the process of making (e.g., Docker images, cacheing)?

[yzhang-23]

I am probably missing something, but what are the major advantages/disadvantages of this approach. I see the major one of not breaking downstream repos in the master, but won't they still break in dev? Won't this still require a lot of management of keeping dev and master up to date with each other? I do see potential advantages in the testing time, so that would be useful.

@twindus I could be wrong, but my impression is that the downstream repos might be broken in the dev branch, and although testing the dev branch could be done automatically, merging it to master may still take some manual effort, since it may take several cycles of debugging/testing until all longer (regression) tests pass in the end. In my mind, the advantage of this dev/prod model is that we can always have a stable master version of the code to release, the price to pay is that we need to more work in merging dev to master.