sig-testing: alpha/beta "enabled" and "conformance" presubmits

[pohly]

We've had kind-alpha-beta-features and kind-beta-features jobs for a while. The original purpose was to run stable tests in a cluster with features enabled to detect when enabling those breaks stable functionality. Later the jobs were extended to also run all tests which should work in such a cluster. The kind-alpha-features job got removed recently because it's not necessarily a valid cluster configuration.

What this adds for the both cluster configs is:

• "-enabled": running only tests for on-by-default features, i.e. excluding tests for stable features. This matches the original purpose of the jobs.
• "-enabled-conformance": restricts the test selection even further to only conformance tests.

Both can eventually get promoted to release informing or even blocking.

Only presubmits get added for now. If testing the jobs in a trial PR works, the corresponding periodics can be added.

For reference:

• label filtering: https://docs.google.com/document/d/1YkU7EusfOZO9jgJbHvCwMCfMZ5CxYMxZHAblpDdKbUQ/edit?tab=t.0#heading=h.kvj5zur6ebww
• job naming discussion: https://kubernetes.slack.com/archives/C09QZ4DQB/p1749500650854679?thread_ts=1748885684.963169&cid=C09QZ4DQB

[BenTheElder]

/hold

AllAlpha=true without a corresponding AllBeta=true is not guaranteed to work because an alpha feature might have a dependency on an off-by-default beta feature.

For testing alpha features, sure. But we can cover that by exposing that the test has such dependencies and skipping them.

For starting a cluster, it should absolutely be safe to set AllAlpha, not use those features, and have GA / default features work fine.

[BenTheElder]

we skip BetaOffByDefault here so that we do not need to remove this job

[pohly]

The problem is not which tests we run, it is the cluster config itself (see kubernetes/kubernetes#133697 (comment) and following).

[BenTheElder]

I see this discussion now: kubernetes/kubernetes#133697 (comment)

I think we should retain "only test default features (which should be stable), but enable alphas (which shouldn't destabilize the cluster if unused)", but I'm not sure what to call that job.

It will instead be "enable all features".

[pohly]

I think we should retain "only test default features (which should be stable), but enable alphas (which shouldn't destabilize the cluster if unused)", but I'm not sure what to call that job.

Isn't that the "alpha-beta-enabled" variant that we discussed in https://kubernetes.slack.com/archives/C09QZ4DQB/p1749500650854679?thread_ts=1748885684.963169&cid=C09QZ4DQB? We never got around to adding those. I didn't do it at that time and then had the impression that you were working on it together with some communication to the project about this.

We can also do "beta-enabled". We can't do "alpha-enabled", for the same reason that we cannot do "alpha-features" (potentially invalid cluster config).

[BenTheElder]

We can also do "beta-enabled". We can't do "alpha-enabled", for the same reason that we cannot do "alpha-features" (potentially invalid cluster config).

yeah. I think the most value will be the alpha-beta-enabled but we should also do beta-enabled.

Also commented back on the main thread, I don't know what to call it without being a mouthful but something like alpha-beta-conformance in the way we have ga-only conformance, I think breaking conformance tests by merely enabling these is a red flag and maybe that's a better target for a periodic than just alpha-enabled + default presubmits

[pohly]

The difference between alpha-beta-enabled would be that it runs all GA tests, whereas alpha-beta-conformance is limited to conformance tests, right?

We can do both, but I am not sure how useful alpha-beta-conformance is, assuming that we also monitor alpha-beta-enabled breakage. Both depend on alpha-level functionality, even if they don't test it, and thus cannot become release blocking. They can both be release informing, and then alpha-beta-conformance is redundant.

Coming back to this PR: can we merge it now and then add the missing jobs separately, or do you want me to turn this PR into the overall "clean up feature gate testing jobs" PR?

[BenTheElder]

The difference between alpha-beta-enabled would be that it runs all GA tests, whereas alpha-beta-conformance is limited to conformance tests, right?

s/GA/on-by-default/g, and not all, typically we would be skipping serial and slow for example, whereas conformance does not. they have overlap.

We can do both, but I am not sure how useful alpha-beta-conformance is, assuming that we also monitor alpha-beta-enabled breakage. Both depend on alpha-level functionality, even if they don't test it, and thus cannot become release blocking.

We have no rule that says this. And anyhow, enabling it should be safe.

Regardless, new jobs must start by proposing to informing, not blocking yet.

They can both be release informing, and then alpha-beta-conformance is redundant.

I don't follow why it's redundant, conformance is largely a superset of the tests we run in presubmit. It may make more sense to just have the conformance version though.

[aojea]

oh, I totally forgot that to do alpha, beta feature jobs we also added the capacity to run the e2e tests with that specific labels.
Having the twin job running only conformance has demonstrated to be very efficient to discard issues of stability caused by bad tests for helping release-signal folks to correlate issues, i.e. if kind-alpha-beta fails constantly but kind-alpha-beta-conformance does not, it has to be something related to some specific test and not the job

[pohly]

s/GA/on-by-default/g, and not all, typically we would be skipping serial and slow for example,

Ack to on-by-default. That typically means tests for beta features, and with "beta is the new GA" that means we would run tests that really should pass. Test failures in "alpha-beta-enabled" (all features enabled, run on-by-default tests) need the same attention as failures in "alpha-beta-conformance" (all features enabled, run only conformance tests).

I don't see why we should skip serial or slow in an "alpha-beta-enabled" CI job. If we have such tests, we should run them. I would also include them in an optional presubmit. The only situation where excluding them may make sense is in a presubmit which always runs, to keep resource usage lower and to avoid delays.

Having the twin job running only conformance has demonstrated to be very efficient to discard issues of stability caused by bad tests for helping release-signal folks to correlate issues

Providing "alpha-beta-conformance" as a slightly more stable (because less tests) version of "alpha-beta-enabled" for SIG release may make sense. It all depends on who is going to monitor those jobs.

So shall I include the new jobs here or not?

[pohly]

So shall I include the new jobs here or not?

The kind-alpha-features jobs got removed through a different PR. As we started the discussion around them (again...) here, let's repurpose this PR to add the missing jobs. I've started with presubmits for now and will do periodics next if this works out okay.

[BenTheElder]

/lgtm

[pohly]

I made the same change as in #35575 (disable the SKIP default): https://github.com/kubernetes/test-infra/compare/212dcd7c76b7f31e94c8614c7742abae45d63829..04f210d27d9f0c864998a8d4b2d804dc68be8bdf

@aojea: can you perhaps LGTM + approve both this and the other PR? I'd like to make some progress on this.

[aojea]

absolutely
/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea, pohly

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/OWNERS [aojea]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[aojea]

/hold cancel

[k8s-ci-robot]

@pohly: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

• key kubernetes-kind-presubmits.yaml using file config/jobs/kubernetes/sig-testing/kubernetes-kind-presubmits.yaml

In response to this:

We've had kind-alpha-beta-features and kind-beta-features jobs for a while. The original purpose was to run stable tests in a cluster with features enabled to detect when enabling those breaks stable functionality. Later the jobs were extended to also run all tests which should work in such a cluster. The kind-alpha-features job got removed recently because it's not necessarily a valid cluster configuration.

What this adds for the both cluster configs is:

• "-enabled": running only tests for on-by-default features, i.e. excluding tests for stable features. This matches the original purpose of the jobs.
• "-enabled-conformance": restricts the test selection even further to only conformance tests.

Both can eventually get promoted to release informing or even blocking.

Only presubmits get added for now. If testing the jobs in a trial PR works, the corresponding periodics can be added.

For reference:

• label filtering: https://docs.google.com/document/d/1YkU7EusfOZO9jgJbHvCwMCfMZ5CxYMxZHAblpDdKbUQ/edit?tab=t.0#heading=h.kvj5zur6ebww
• job naming discussion: https://kubernetes.slack.com/archives/C09QZ4DQB/p1749500650854679?thread_ts=1748885684.963169&cid=C09QZ4DQB

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.