OCPQE-30739:Support multi-substring pattern matching for TestGrid #4773

kuiwang02 · 2025-10-20T04:09:29Z

Refine TestGrid job filtering for operator-framework jobs

Problem

Previously, the IsSpecialInformingJobOnTestGrid function used a simple prefix-based matching approach. This meant that all jobs starting with a specific prefix (e.g., periodic-ci-openshift-operator-framework-operator-controller- or periodic-ci-openshift-operator-framework-olm-release-) would automatically appear in TestGrid.

However, not all jobs in a repository should appear in TestGrid immediately. Some jobs are still stabilizing and should only exist in Sippy as candidate job tier for monitoring purposes. Only stable, production-ready jobs (typically those with default or stable in their names) should be visible in TestGrid and classified as informing jobs.

Solution

This PR refines the filtering logic by introducing a flexible pattern-matching system inspired by the setJobTier function in sippy/pkg/variantregistry/ocp.go:

Changed testGridInformingPrefixes from []string to [][]string
Each pattern entry is now a list of substrings:
- First element: Must match as a prefix (using strings.HasPrefix)
- Remaining elements: Must all be present as substrings (using strings.Contains)
Extracted a helper function matchesAllSubstrings for cleaner code organization

Changes for operator-framework jobs

For operator-framework repositories, this PR adds selective filtering:

Before:

{"periodic-ci-openshift-operator-framework-olm-release-"},
{"periodic-ci-openshift-operator-framework-operator-controller-"},
All jobs with these prefixes appeared in TestGrid.

After:
{"periodic-ci-openshift-operator-framework-olm-release-", "default"},
{"periodic-ci-openshift-operator-framework-olm-release-", "stable"},
{"periodic-ci-openshift-operator-framework-operator-controller-", "default"},
{"periodic-ci-openshift-operator-framework-operator-controller-", "stable"},
Only jobs containing default or stable in their names appear in TestGrid.

Example

For the openshift-operator-framework-operator-controller-release-4.21 periodics:

Job Name	In TestGrid?	In Sippy?	Job Tier
periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-default-catalog-consistency	✅ Yes	✅ Yes	Informing
periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-stable-test	✅ Yes	✅ Yes	Informing
periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-extended-f3	❌ No	✅ Yes	Candidate

Jobs without default or stable remain in Sippy as candidate jobs for monitoring but won't clutter TestGrid dashboards. Once these jobs stabilize, teams can rename them to include default or stable, automatically promoting them to TestGrid visibility and informing tier.

Benefits

Gradual stabilization: New jobs can be monitored in Sippy without appearing in TestGrid
Name-based control: Teams control TestGrid visibility through job naming conventions
Flexible pattern matching: The [][]string structure supports complex matching rules
Backward compatible: Existing jobs continue to work unchanged

Testing

Comprehensive unit tests were added to verify the behavior:

  package util

  import "testing"

  func TestIsSpecialInformingJobOnTestGrid(t *testing.T) {
        testCases := []struct {
                name     string
                jobName  string
                expected bool
        }{
                // Standard informing jobs that should always be included
                {
                        name:     "openshift-release-master-nightly job",
                        jobName:  "periodic-ci-openshift-release-master-nightly-4.21-e2e-aws-ovn",
                        expected: true,
                },
                {
                        name:     "openshift-hypershift job",
                        jobName:  "periodic-ci-openshift-hypershift-main-periodics-e2e-aws-ovn",
                        expected: true,
                },
                // Operator Framework OLM jobs - only with "default" or "stable"
                {
                        name:     "olm job with default",
                        jobName:  "periodic-ci-openshift-operator-framework-olm-release-4.21-periodics-e2e-aws-ovn-default",
                        expected: true,
                },
                {
                        name:     "olm job with stable",
                        jobName:  "periodic-ci-openshift-operator-framework-olm-release-4.21-periodics-e2e-aws-ovn-stable",
                        expected: true,
                },
                {
                        name:     "olm job without default or stable",
                        jobName:  "periodic-ci-openshift-operator-framework-olm-release-4.21-periodics-e2e-aws-ovn-extended-f3",
                        expected: false,
                },
                {
                        name:     "olm job with flaky",
                        jobName:  "periodic-ci-openshift-operator-framework-olm-release-4.21-periodics-e2e-gcp-olm-flaky",
                        expected: false,
                },
                // Operator Framework operator-controller jobs - only with "default" or "stable"
                {
                        name:     "operator-controller job with default",
                        jobName:  "periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-default-catalog-consistency",
                        expected: true,
                },
                {
                        name:     "operator-controller job with stable",
                        jobName:  "periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-stable-test",
                        expected: true,
                },
                {
                        name:     "operator-controller job without default or stable",
                        jobName:  "periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-extended-f3",
                        expected: false,
                },
                // Edge cases
                {
                        name:     "olm job with 'default' in middle of name",
                        jobName:  "periodic-ci-openshift-operator-framework-olm-release-4.21-periodics-e2e-aws-default-ovn",
                        expected: true,
                },
                {
                        name:     "random job not matching any prefix",
                        jobName:  "periodic-ci-some-random-repo-main-e2e",
                        expected: false,
                },
        }

        for _, tc := range testCases {
                t.Run(tc.name, func(t *testing.T) {
                        result := IsSpecialInformingJobOnTestGrid(tc.jobName)
                        if result != tc.expected {
                                t.Errorf("IsSpecialInformingJobOnTestGrid(%q) = %v, expected %v", tc.jobName, result, tc.expected)
                        }
                })
        }
  }

  $ go test -v ./pkg/util -run TestIsSpecialInformingJobOnTestGrid
  === RUN   TestIsSpecialInformingJobOnTestGrid
  === RUN   TestIsSpecialInformingJobOnTestGrid/openshift-release-master-nightly_job
  === RUN   TestIsSpecialInformingJobOnTestGrid/openshift-hypershift_job
  === RUN   TestIsSpecialInformingJobOnTestGrid/olm_job_with_default
  === RUN   TestIsSpecialInformingJobOnTestGrid/olm_job_with_stable
  === RUN   TestIsSpecialInformingJobOnTestGrid/olm_job_without_default_or_stable
  === RUN   TestIsSpecialInformingJobOnTestGrid/olm_job_with_flaky
  === RUN   TestIsSpecialInformingJobOnTestGrid/operator-controller_job_with_default
  === RUN   TestIsSpecialInformingJobOnTestGrid/operator-controller_job_with_stable
  === RUN   TestIsSpecialInformingJobOnTestGrid/operator-controller_job_without_default_or_stable
  === RUN   TestIsSpecialInformingJobOnTestGrid/olm_job_with_'default'_in_middle_of_name
  === RUN   TestIsSpecialInformingJobOnTestGrid/random_job_not_matching_any_prefix
  --- PASS: TestIsSpecialInformingJobOnTestGrid (0.00s)
      --- PASS: TestIsSpecialInformingJobOnTestGrid/openshift-release-master-nightly_job (0.00s)
      --- PASS: TestIsSpecialInformingJobOnTestGrid/openshift-hypershift_job (0.00s)
      --- PASS: TestIsSpecialInformingJobOnTestGrid/olm_job_with_default (0.00s)
      --- PASS: TestIsSpecialInformingJobOnTestGrid/olm_job_with_stable (0.00s)
      --- PASS: TestIsSpecialInformingJobOnTestGrid/olm_job_without_default_or_stable (0.00s)
      --- PASS: TestIsSpecialInformingJobOnTestGrid/olm_job_with_flaky (0.00s)
      --- PASS: TestIsSpecialInformingJobOnTestGrid/operator-controller_job_with_default (0.00s)
      --- PASS: TestIsSpecialInformingJobOnTestGrid/operator-controller_job_with_stable (0.00s)
      --- PASS: TestIsSpecialInformingJobOnTestGrid/operator-controller_job_without_default_or_stable (0.00s)
      --- PASS: TestIsSpecialInformingJobOnTestGrid/olm_job_with_'default'_in_middle_of_name (0.00s)
      --- PASS: TestIsSpecialInformingJobOnTestGrid/random_job_not_matching_any_prefix (0.00s)
  PASS
  ok      github.com/openshift/ci-tools/pkg/util 0.949s

Test coverage includes:

✅ Standard informing jobs (release-master-nightly, hypershift) - should always match
✅ OLM jobs with default - should match
✅ OLM jobs with stable - should match
❌ OLM jobs with flaky or extended-f3 - should NOT match
✅ operator-controller jobs with default or stable - should match
❌ operator-controller jobs without qualifying keywords - should NOT match
✅ Edge cases like default appearing in the middle of job names - should match
❌ Random jobs not matching any pattern - should NOT match

Assisted-by: Claude Code

…stGrid

kuiwang02 · 2025-10-20T04:11:06Z

/retitle "OCPQE-30739:Support multi-substring pattern matching for TestGrid"

openshift-ci-robot · 2025-10-20T04:11:14Z

@kuiwang02: This pull request references OCPQE-30739 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.21.0" version, but no target version was set.

In response to this:

Refine TestGrid job filtering for operator-framework jobs

Problem

Previously, the IsSpecialInformingJobOnTestGrid function used a simple prefix-based matching approach. This meant that all jobs starting with a specific prefix (e.g., periodic-ci-openshift-operator-framework-operator-controller- or periodic-ci-openshift-operator-framework-olm-release-) would automatically appear in TestGrid.

However, not all jobs in a repository should appear in TestGrid immediately. Some jobs are still stabilizing and should only exist in Sippy as candidate job tier for monitoring purposes. Only stable, production-ready jobs (typically those with default or stable in their names) should be visible in TestGrid and classified as informing jobs.

Solution

This PR refines the filtering logic by introducing a flexible pattern-matching system inspired by the setJobTier function in sippy/pkg/variantregistry/ocp.go:

Changed testGridInformingPrefixes from []string to [][]string
Each pattern entry is now a list of substrings:
- First element: Must match as a prefix (using strings.HasPrefix)
- Remaining elements: Must all be present as substrings (using strings.Contains)
Extracted a helper function matchesAllSubstrings for cleaner code organization

Changes for operator-framework jobs

For operator-framework repositories, this PR adds selective filtering:

Before:

{"periodic-ci-openshift-operator-framework-olm-release-"},
{"periodic-ci-openshift-operator-framework-operator-controller-"},
All jobs with these prefixes appeared in TestGrid.

After:
{"periodic-ci-openshift-operator-framework-olm-release-", "default"},
{"periodic-ci-openshift-operator-framework-olm-release-", "stable"},
{"periodic-ci-openshift-operator-framework-operator-controller-", "default"},
{"periodic-ci-openshift-operator-framework-operator-controller-", "stable"},
Only jobs containing default or stable in their names appear in TestGrid.

Example

For the openshift-operator-framework-operator-controller-release-4.21 periodics:

Job Name	In TestGrid?	In Sippy?	Job Tier
periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-default-catalog-consistency	✅ Yes	✅ Yes	Informing
periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-stable-test	✅ Yes	✅ Yes	Informing
periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-extended-f3	❌ No	✅ Yes	Candidate

Jobs without default or stable remain in Sippy as candidate jobs for monitoring but won't clutter TestGrid dashboards. Once these jobs stabilize, teams can rename them to include default or stable, automatically promoting them to TestGrid visibility and informing tier.

Benefits

Gradual stabilization: New jobs can be monitored in Sippy without appearing in TestGrid
Name-based control: Teams control TestGrid visibility through job naming conventions
Flexible pattern matching: The [][]string structure supports complex matching rules
Backward compatible: Existing jobs continue to work unchanged

Testing

Comprehensive unit tests were added to verify the behavior:

 package util

 import "testing"

 func TestIsSpecialInformingJobOnTestGrid(t *testing.T) {
       testCases := []struct {
               name     string
               jobName  string
               expected bool
       }{
               // Standard informing jobs that should always be included
               {
                       name:     "openshift-release-master-nightly job",
                       jobName:  "periodic-ci-openshift-release-master-nightly-4.21-e2e-aws-ovn",
                       expected: true,
               },
               {
                       name:     "openshift-hypershift job",
                       jobName:  "periodic-ci-openshift-hypershift-main-periodics-e2e-aws-ovn",
                       expected: true,
               },
               // Operator Framework OLM jobs - only with "default" or "stable"
               {
                       name:     "olm job with default",
                       jobName:  "periodic-ci-openshift-operator-framework-olm-release-4.21-periodics-e2e-aws-ovn-default",
                       expected: true,
               },
               {
                       name:     "olm job with stable",
                       jobName:  "periodic-ci-openshift-operator-framework-olm-release-4.21-periodics-e2e-aws-ovn-stable",
                       expected: true,
               },
               {
                       name:     "olm job without default or stable",
                       jobName:  "periodic-ci-openshift-operator-framework-olm-release-4.21-periodics-e2e-aws-ovn-extended-f3",
                       expected: false,
               },
               {
                       name:     "olm job with flaky",
                       jobName:  "periodic-ci-openshift-operator-framework-olm-release-4.21-periodics-e2e-gcp-olm-flaky",
                       expected: false,
               },
               // Operator Framework operator-controller jobs - only with "default" or "stable"
               {
                       name:     "operator-controller job with default",
                       jobName:  "periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-default-catalog-consistency",
                       expected: true,
               },
               {
                       name:     "operator-controller job with stable",
                       jobName:  "periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-stable-test",
                       expected: true,
               },
               {
                       name:     "operator-controller job without default or stable",
                       jobName:  "periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-extended-f3",
                       expected: false,
               },
               // Edge cases
               {
                       name:     "olm job with 'default' in middle of name",
                       jobName:  "periodic-ci-openshift-operator-framework-olm-release-4.21-periodics-e2e-aws-default-ovn",
                       expected: true,
               },
               {
                       name:     "random job not matching any prefix",
                       jobName:  "periodic-ci-some-random-repo-main-e2e",
                       expected: false,
               },
       }

       for _, tc := range testCases {
               t.Run(tc.name, func(t *testing.T) {
                       result := IsSpecialInformingJobOnTestGrid(tc.jobName)
                       if result != tc.expected {
                               t.Errorf("IsSpecialInformingJobOnTestGrid(%q) = %v, expected %v", tc.jobName, result, tc.expected)
                       }
               })
       }
 }

 $ go test -v ./pkg/util -run TestIsSpecialInformingJobOnTestGrid
 === RUN   TestIsSpecialInformingJobOnTestGrid
 === RUN   TestIsSpecialInformingJobOnTestGrid/openshift-release-master-nightly_job
 === RUN   TestIsSpecialInformingJobOnTestGrid/openshift-hypershift_job
 === RUN   TestIsSpecialInformingJobOnTestGrid/olm_job_with_default
 === RUN   TestIsSpecialInformingJobOnTestGrid/olm_job_with_stable
 === RUN   TestIsSpecialInformingJobOnTestGrid/olm_job_without_default_or_stable
 === RUN   TestIsSpecialInformingJobOnTestGrid/olm_job_with_flaky
 === RUN   TestIsSpecialInformingJobOnTestGrid/operator-controller_job_with_default
 === RUN   TestIsSpecialInformingJobOnTestGrid/operator-controller_job_with_stable
 === RUN   TestIsSpecialInformingJobOnTestGrid/operator-controller_job_without_default_or_stable
 === RUN   TestIsSpecialInformingJobOnTestGrid/olm_job_with_'default'_in_middle_of_name
 === RUN   TestIsSpecialInformingJobOnTestGrid/random_job_not_matching_any_prefix
 --- PASS: TestIsSpecialInformingJobOnTestGrid (0.00s)
     --- PASS: TestIsSpecialInformingJobOnTestGrid/openshift-release-master-nightly_job (0.00s)
     --- PASS: TestIsSpecialInformingJobOnTestGrid/openshift-hypershift_job (0.00s)
     --- PASS: TestIsSpecialInformingJobOnTestGrid/olm_job_with_default (0.00s)
     --- PASS: TestIsSpecialInformingJobOnTestGrid/olm_job_with_stable (0.00s)
     --- PASS: TestIsSpecialInformingJobOnTestGrid/olm_job_without_default_or_stable (0.00s)
     --- PASS: TestIsSpecialInformingJobOnTestGrid/olm_job_with_flaky (0.00s)
     --- PASS: TestIsSpecialInformingJobOnTestGrid/operator-controller_job_with_default (0.00s)
     --- PASS: TestIsSpecialInformingJobOnTestGrid/operator-controller_job_with_stable (0.00s)
     --- PASS: TestIsSpecialInformingJobOnTestGrid/operator-controller_job_without_default_or_stable (0.00s)
     --- PASS: TestIsSpecialInformingJobOnTestGrid/olm_job_with_'default'_in_middle_of_name (0.00s)
     --- PASS: TestIsSpecialInformingJobOnTestGrid/random_job_not_matching_any_prefix (0.00s)
 PASS
 ok      github.com/openshift/ci-tools/pkg/util 0.949s

Test coverage includes:

✅ Standard informing jobs (release-master-nightly, hypershift) - should always match
✅ OLM jobs with default - should match
✅ OLM jobs with stable - should match
❌ OLM jobs with flaky or extended-f3 - should NOT match
✅ operator-controller jobs with default or stable - should match
❌ operator-controller jobs without qualifying keywords - should NOT match
✅ Edge cases like default appearing in the middle of job names - should match
❌ Random jobs not matching any pattern - should NOT match

Assisted-by: Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

kuiwang02 · 2025-10-20T04:11:43Z

/retitle OCPQE-30739:Support multi-substring pattern matching for TestGrid

kuiwang02 · 2025-10-20T05:41:36Z

/retest-required

jianzhangbjz · 2025-10-20T06:11:32Z

/lgtm

openshift-ci-robot · 2025-10-20T06:12:29Z

Scheduling required tests:
/test e2e

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test integration-optional-test

openshift-ci · 2025-10-20T06:12:40Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jianzhangbjz, kuiwang02
Once this PR has been reviewed and has the lgtm label, please assign pruan-rht for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kuiwang02 · 2025-10-20T06:53:12Z

@bear-redhat could you please review it? if it is ok, could you please approve it? thanks

kuiwang02 · 2025-10-20T08:20:46Z

/retest-required

kuiwang02 · 2025-10-20T23:58:50Z

/retest-required

openshift-ci · 2025-10-21T02:18:51Z

@kuiwang02: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/breaking-changes	`d3897ae`	link	false	`/test breaking-changes`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

kuiwang02 · 2025-10-21T03:08:45Z

/retest-required

kuiwang02 · 2025-10-21T09:23:53Z

@bear-redhat could you please review it? if it is ok, could you please approve it? thanks

Support multi-substring pattern matching in IsSpecialInformingJobOnTe…

d3897ae

…stGrid

openshift-ci bot requested review from bear-redhat and psalajova October 20, 2025 04:09

openshift-ci bot changed the title ~~Support multi-substring pattern matching in IsSpecialInformingJobOnTe…~~ "OCPQE-30739:Support multi-substring pattern matching for TestGrid" Oct 20, 2025

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Oct 20, 2025

openshift-ci bot changed the title ~~"OCPQE-30739:Support multi-substring pattern matching for TestGrid"~~ OCPQE-30739:Support multi-substring pattern matching for TestGrid Oct 20, 2025

openshift-ci bot assigned jianzhangbjz Oct 20, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 20, 2025

OCPQE-30739:Support multi-substring pattern matching for TestGrid #4773

Are you sure you want to change the base?

OCPQE-30739:Support multi-substring pattern matching for TestGrid #4773

Uh oh!

Conversation

kuiwang02 commented Oct 20, 2025

Refine TestGrid job filtering for operator-framework jobs

Problem

Solution

Changes for operator-framework jobs

Uh oh!

kuiwang02 commented Oct 20, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Oct 20, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Refine TestGrid job filtering for operator-framework jobs

Problem

Solution

Changes for operator-framework jobs

Uh oh!

kuiwang02 commented Oct 20, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kuiwang02 commented Oct 20, 2025

Uh oh!

jianzhangbjz commented Oct 20, 2025

Uh oh!

openshift-ci-robot commented Oct 20, 2025

Uh oh!

openshift-ci bot commented Oct 20, 2025

Uh oh!

kuiwang02 commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kuiwang02 commented Oct 20, 2025

Uh oh!

kuiwang02 commented Oct 20, 2025

Uh oh!

openshift-ci bot commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kuiwang02 commented Oct 21, 2025

Uh oh!

kuiwang02 commented Oct 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kuiwang02 commented Oct 20, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Oct 20, 2025 •

edited by openshift-ci bot

Loading

kuiwang02 commented Oct 20, 2025 •

edited by openshift-ci bot

Loading

kuiwang02 commented Oct 20, 2025 •

edited

Loading

openshift-ci bot commented Oct 21, 2025 •

edited

Loading