Clarify graduation submission requirements/review

[duglin]

As I look at the graduation process for CloudEvents and KEDA it appears that the current process might need a bit of tweaking to help speed things up. Some observations/suggestions, from an outsider's point of view:

• not everything that is required of a project is documented. While there are some (what I would call) house-keeping requirements (like a COC, best practices badges), it's not clear what about the project and its community make it worthy of being "graduated". The graduation template does ask "why do you think you're ready?", but there's little in the way of guidance to indicate what the minimum bars are to to be approved. As a result, my impression is that it feels more like a "gut feel" of the TOC more than anything else. Granted, in the end that might be what it needs to be since having hard metrics to cover both "explosive growth projects" as well as "mature, slow moving but valued projects" might not be possible. However, we should document those cases where firm requirements are known (e.g. at least 2 orgs have maintainers), and then simply ask for the stats for the other metrics of interest. Either way, any data that the TOC uses should be documented as part of the graduation application. Very related to the discussion in Add clarification to CNCF graduation criteria #992 .
• the TOC members are busy and therefore are the bottleneck when it comes to preparing the data that is to be reviewed. I'm proposing that we shift that burden to the project itself. To go along with the previous bullet, the graduation application should include all information the TOC members would like to review - then it's up to the project to provide that information. This shift the burden of the TOC (or the single TOC sponsor) from information gathers to application/information reviewer. If more information is needed then the TOC (or any other CNCF memebr) can ask for it via comments on the application PR.

To that end, this PR proposes a change to the graduation process:

• each graduation application becomes a PR to add the project's application to the github repo
• TOC/SIGs/TAGs/CNCF members... can then review and ask for additional information to be added if needed
• once all information is provided, TOC members simply LGTM the PR and when 6 or more have done so it can then be added to the next TOC call for a formal vote. Unless people are comfortable with the LGTM/async voting process - which would be ideal.

My ask, if this direction seems reasonable, is that the TOC members propose additional items to the application template in this PR so that it eventually includes all information they will need to make their decision. I believe that aside from speeding up the process, it will also increase the transparency of the the process.

/cc @tomkerkhove

[duglin]

It's worth noting that I think a similar process can be used for sandbox and incubator proposal, in particular when it comes to the DD analysis as I think that's probably one of the bigger work items in the process. Shift that to the project to gather.

[tomkerkhove]

Great addition!

[tomkerkhove]

Or should we point to the maintainer overview of the given project to avoid stale information? In theory this should be well-documented for a graduation project

[duglin]

Good idea - I decided to do both... URL to maintainer doc AND the current list of orgs who have maintainers. I think having today's # of orgs is a good bit of info for the TOC to have w/o having to follow a link to see it.

[tomkerkhove]

We should also add links to the required reviews, if any are mandatory

[duglin]

I modified the security audit bullet to make it so we can add more reviews under there as needed - then they're all in one spot. Are there other reviews I should add?

[tomkerkhove]

• TAG security was once required (but turned out external audit was enough) - Proposal for KEDA to become a CNCF Graduated project #925 (comment)
• TAG Contributor strategy was required for KEDA - Proposal for KEDA to become a CNCF Graduated project #925 (comment)

[duglin]

for TAG Contrib... let's see if we can create the list of requirements that the TAG used and then add them to this doc/PR. I've added a commit with some stuff from KEDA's review - see if anything is missing.

[tomkerkhove]

Adding @craigbox as we had similar chats recently for Istio

[craigbox]

I disagree with the need for any due diligence at the graduation phase, unless there is any suggestion that a project may have regressed since entering the CNCF and may not continue to meet the incubation criteria. That process could well happen with an incubating project and should be addressed by project review and/or the archival process, and need not be duplicated by graduation.

I take this position based on the expressed ideals that "[the] incubation level is the point at which we expect to perform full due diligence on projects", and that the stated intention of graduation is a signal to the market that the software is ready for the early majority, not just early adopters.

An exception would be that if a project intends to join at Graduated, which has not yet happened but is explicitly mentioned as a possibility. In that case, I would hope that the processes for "join at Incubation" and "demonstrate readiness for Graduation" were non-duplicative and could just both be followed.

[duglin]

I'm ok with removing it if the TOC would like that. I included it because:
1 - as you suggested, a project may want to join the CNCF at the graduated level
2 - since part of my goal is to reduce the amount of time for the TOC to review any pertinent information about a project, I wanted to put everything in one spot, even if that info was originally provided at a lower stage.
3 - worst case people could just copy the info from the info previously provided

I'll leave it to the TOC to decide if including this here is useful or not - I'll do whatever speeds up their graduation review process.

[duglin]

I should point out that ideally I'd like to remove, not add, questions to the form that applicants need to fill out because it seems like many of the questions (while interesting) should not be used to determine if a project is "graduation worthy". I say this because the answers are often transient or misleading. For example, "commit stats". For a younger project there could look really good, but once the project is very mature it may not look as good. But that doesn't mean the project isn't worthy... it could just mean it's stable. And in fact, too many commits could be a sign of being buggy. Plus, would a project lose it's "graduated status" if the answers to those questions suddenly changed? I doubt it. So, those questions shouldn't really be used to gauge graduation readiness.

And while I'm at it... there's part of me that even questions the need for the 3 levels at all since projects don't seem to be able to be demoted if things change (beyond being archived). Why should a once-good-but-now-not-so-good project remain at "graduated" status while newer projects that are wildly popular might be stuck at "incubator" stage? The status doesn't really provide the true value to the community who might view it as a way to gauge a project's worthiness. It's potentially very misleading and will only get worse as the entire Cloud Native community matures and project slowly fad in popularity. As a result, I often wonder if there should only be one "bar"... either a project is a CNCF project or not. A project meets certain criteria (probably similar to what we have for incubator) and once they meet that then they're in. Very low bar and no more politics or games to worry about... just focus on the project being successful. But that might be a topic for a different day...

[craigbox]

PTAL at #992 and #997. @duglin, it might be a good idea to attend the upcoming meeting if you can.

I think I'm in favour of your bar criteria, with

• Associate CNCF Project (what is Sandbox today - I would call this "Incubating" but I wouldn't want to re-use the word)
• CNCF Project (what is Incubation today)

and then a set of badges which can be renewed semi-frequently based on "number of companies contributing", "diversity of governance", "when the last security audit was done", etc.

[TheFoxAtWork]

@duglin we missed you at the task force presentation to the TOC today. Given your involvement with the group and the consensus the group achieved, do you believe this PR to be overcome by the group's deliverable/recommendations? I realize the work is not "complete" in that the changes are not reflected in the TOC repo.

[duglin]

As I'm not involved in the group any more I'll have to wait until the TOC repo changes are made before I can say for sure. The main things I'll be looking for are:

• near zero back-n-forth between project and CNCF. Ideally, a process like: project submits a PR and TOC votes.
• a single list of things that the project needs to do, or verify, to be approved - with results being part of the PR. Ideally a list in a single location and not spread out across multiple docs

[TheFoxAtWork]

That is reasonable. We'll leave this PR open, and I'll try to ensure those points you highlighted are covered or addressed by the resulting consensus reflected in the PR(s) to document those changes.

[TheFoxAtWork]

Closing this PR as overcome by the recent changes to the TOC repo. Related issue to capture project feedback on this process: #1293