feat(syncwaves): add DAG for syncwaves (alpha) #7437

[SebastienFelix]

This PR resolves #7437

This PR introduces a new concept to sync-waves : sync-groups, which allows to define groups of resources that could be synced independently using a DAG representing their dependencies.

To give some insights of what I am trying to achieve with this PR, I add a new layer to sync-waves in to order to define groups of resources that could be synced independently (in parallel) using sync waves. By adding a wave-group id and wave group dependencies, it allows to define a DAG operating on the different wave groups. Here is an example of resource definition and the expected behavior :

apiVersion: v1
kind: Pod
metadata:
name: label-demo
labels:
argocd.argoproj.io/sync-wave: -1
argocd.argoproj.io/sync-wave-group: 2
argocd.argoproj.io/sync-wave-group-dependencies: 0,1

This pod belongs to wave-group 2 with sync-wave value -1. It will be synced only when no more resources from sync-groups 0 and 1 needs to be synced.

How Does It Work Together?

Syncing process orders the resources in the following precedence:

The phase
The group with respect to group dependencies
The wave they are in (lower values first)
By kind (e.g. namespaces first)
By name

Hope it clarifies.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Title of the PR
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

[bunnyshell]

🔴 Preview Environment stopped on Bunnyshell

See: Environment Details | Pipeline Logs

Available commands (reply to this comment):

• 🔵 /bns:start to start the environment
• 🚀 /bns:deploy to redeploy the environment
• ❌ /bns:delete to remove the environment

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.81%. Comparing base (b74cf45) to head (b7251e2).

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #24935      +/-   ##
==========================================
- Coverage   60.82%   60.81%   -0.01%     
==========================================
  Files         351      351              
  Lines       60439    60439              
==========================================
- Hits        36760    36754       -6     
- Misses      20757    20771      +14     
+ Partials     2922     2914       -8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[todaywasawesome]

Thanks @SebastienFelix - what I'm struggling to understand here is how this relates to app dependency management, which is a problem between applications rather than resources in the same application. I understand this PR to deal with resources within a single application.

Secondly, what does adding a dependency give me that sync waves do not already provide? Sync-wave: 2 has an inherent dependency that all syncwaves lower than 2 have completed. This PR gives me the ability to have two sync waves run in parallel but aren't all resources inside a sync wave basically applied at the same time anyway?

[SebastienFelix]

Hello @todaywasawesome - I referenced the issue #7437 because my original desire was to tackle the issue described here #3517 .

As mentionned by @crenshaw-dev, the previous issue was closed in favor of #7437. Maybe I have tagged the wrong issue. My apologies if that's the case.

For your second point, I think that the use case described in #3517 explains the need for this feature. Make deployments quicker by avoiding unnecessary waiting time.

Another use-case that could benefit from this feature is the following :

How would you ensure that after a failed sync of a resource from group B, groups C & D would be synced anyway (same apply for a resource from group C and groups B & D,...)?

Hope this clarifies!

EDIT : I salso stumbled upon this monthes ago https://stackoverflow.com/questions/63806794/argocd-sync-waves-between-apps where the use-case of using sync-waves within app-of-apps was said to be possible. Maybe I was mistaken.
Note that the original tentative can be found here argoproj/gitops-engine#744