NO-JIRA: Clarify test names in OTE

This pull add a section to clarify test names in OTE.
It modifies `hack/update-test-metadata.sh` to give a
hint of what to do to fix the issue.

Author: hongkailiu

cmd/cluster-version-operator-tests/README.md

@@ -21,4 +21,40 @@ After [installing-ginkgo](https://onsi.github.io/ginkgo/#installing-ginkgo):
 $ ginkgo ./test/...
 ```
 
-The output looks nicer this way.
+The output looks nicer this way.
+
+
+## Test Names
+
+### Test labels
+
+See [docs](https://github.com/openshift/origin/tree/main/test/extended#test-labels) for details.
+
+### Ownership
+
+* A `[Jira:"Component"]` tag in the test name (e.g., [Jira:"Cluster Version Operator"]) is preferred to claim the ownership of the test. The component comes from [the list of components of Jira Project OCPBUGS](https://issues.redhat.com/projects/OCPBUGS?selectedItem=com.atlassian.jira.jira-projects-plugin:components-page). See [here](https://github.com/openshift/origin/blob/6b584254d53cdd1b5cd6471b69cb7c22f3e28ecd/test/extended/apiserver/rollout.go#L36) for example.
+
+* Or an entry in the [ci-test-mapping](https://github.com/openshift-eng/ci-test-mapping) repository, which maps tests to owning components. See [here](https://github.com/openshift-eng/ci-test-mapping/tree/main/pkg/components/clusterversionoperator) for example.
+
+### Renaming
+
+If a test is renamed, claim it with [specs.Walk](https://pkg.go.dev/github.com/openshift-eng/[email protected]/pkg/extension/extensiontests#ExtensionTestSpecs.Walk):
+
+```go
+specs = specs.Walk(func(spec *et.ExtensionTestSpec) {
+    if orig, ok := renamedTests[spec.Name]; ok {
+		spec.OriginalName = orig
+	}
+})
+```
+
+### Deleting
+
+If a test is renamed, claim it with [ext.IgnoreObsoleteTests](https://pkg.go.dev/github.com/openshift-eng/[email protected]/pkg/extension#Extension.IgnoreObsoleteTests)
+
+```go
+ext.IgnoreObsoleteTests(
+"[sig-abc] My removed test",
+// Add more removed test names below
+)
+```

hack/update-test-metadata.sh

@@ -5,4 +5,8 @@ set -eu
 source hack/build-info.sh
 
 # Update test metadata
-eval "${BIN_PATH}/cluster-version-operator-tests" "update"
+if ! "${BIN_PATH}/cluster-version-operator-tests" "update"; then
+  >&2 echo "run 'make update' to fix the metadata after going through the teest renaming process.
+See https://github.com/openshift/cluster-version-operator/blob/main/cmd/cluster-version-operator-tests/README.md#test-names for details."
+  exit 1
+fi