NO-JIRA: Clarify test names in OTE

This pull add a section to clarify test names in OTE.
It modifies `make update` to with a knob to opt into
deleting the existing metadata.

Author: hongkailiu

Makefile

@@ -29,7 +29,7 @@ integration-test:
 .PHONY: integration-test
 
 update: build
-	hack/update-test-metadata.sh
+	DELETE_EXISTING=TRUE hack/update-test-metadata.sh
 .PHONY: update
 
 format:
@@ -43,7 +43,8 @@ verify-yaml:
 	hack/verify-yaml.sh
 .PHONY: verify-yaml
 
-verify-update: update
+verify-update:
+	hack/update-test-metadata.sh
 	git diff --exit-code HEAD
 .PHONY: verify-update
 

cmd/cluster-version-operator-tests/README.md

@@ -21,4 +21,40 @@ After [installing-ginkgo](https://onsi.github.io/ginkgo/#installing-ginkgo):
 $ ginkgo ./test/...
 ```
 
-The output looks nicer this way.
+The output looks nicer this way.
+
+
+## Test Names
+
+### Test labels
+
+See [docs](https://github.com/openshift/origin/tree/main/test/extended#test-labels) for details.
+
+### Ownership
+
+* A `[Jira:"Component"]` tag in the test name (e.g., [Jira:"Cluster Version Operator"]) is preferred to claim the ownership of the test. The component comes from [the list of components of Jira Project OCPBUGS](https://issues.redhat.com/projects/OCPBUGS?selectedItem=com.atlassian.jira.jira-projects-plugin:components-page). See [here](https://github.com/openshift/origin/blob/6b584254d53cdd1b5cd6471b69cb7c22f3e28ecd/test/extended/apiserver/rollout.go#L36) for example.
+
+* Or an entry in the [ci-test-mapping](https://github.com/openshift-eng/ci-test-mapping) repository, which maps tests to owning components. See [here](https://github.com/openshift-eng/ci-test-mapping/tree/main/pkg/components/clusterversionoperator) for example.
+
+### Renaming
+
+If a test is renamed, claim it with [specs.Walk](https://pkg.go.dev/github.com/openshift-eng/[email protected]/pkg/extension/extensiontests#ExtensionTestSpecs.Walk):
+
+```go
+specs = specs.Walk(func(spec *et.ExtensionTestSpec) {
+    if orig, ok := renamedTests[spec.Name]; ok {
+		spec.OriginalName = orig
+	}
+})
+```
+
+### Deleting
+
+If a test is renamed, claim it with [ext.IgnoreObsoleteTests](https://pkg.go.dev/github.com/openshift-eng/[email protected]/pkg/extension#Extension.IgnoreObsoleteTests)
+
+```go
+ext.IgnoreObsoleteTests(
+"[sig-abc] My removed test",
+// Add more removed test names below
+)
+```

hack/update-test-metadata.sh

@@ -4,5 +4,17 @@ set -eu
 
 source hack/build-info.sh
 
+readonly METADATA_FILE=".openshift-tests-extension/openshift_payload_cluster-version-operator.json"
+DELETE_EXISTING="${DELETE_EXISTING:-false}"
+
+if [[ "${DELETE_EXISTING,,}" == "true" ]] && [[ -f "${METADATA_FILE}" ]] ; then
+  echo "Removing ${METADATA_FILE}"
+  rm .openshift-tests-extension/openshift_payload_cluster-version-operator.json
+fi
+
 # Update test metadata
-eval "${BIN_PATH}/cluster-version-operator-tests" "update"
+if ! "${BIN_PATH}/cluster-version-operator-tests" "update"; then
+  >&2 echo "run 'make update' to fix the metadata
+See https://github.com/openshift/cluster-version-operator/blob/main/cmd/cluster-version-operator-tests/README.md#test-names for details."
+  exit 1
+fi