Add audit annotation for watch rejections due to storage initialization

[ShazaAldawamneh]

What type of PR is this?

What this PR does / why we need it:

When a watch request is rejected because the cacher is still initializing,
the reason was not previously recorded in the audit event. This change:

• Adds a sentinel error ErrStorageInitializing in pkg/storage/cacher/ready.go
• Introduces two new helpers in pkg/apiserver/pkg/audit/context.go:
  • AddAuditAnnotationForRejectWithReason(ctx, reason)
  • AddAuditAnnotationForRejectMessage(ctx, msg)
• Updates waitAndReadGeneration(ctx) to record audit annotations when a watch
  fails due to storage initializing
• Adds a unit test to verify the annotations are correctly added to the audit event

This ensures that audit logs clearly record the reason for rejected watch requests,
improving observability and debug of the API server.

Which issue(s) this PR is related to:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[openshift-ci-robot]

@ShazaAldawamneh: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

• 188e34f|Add audit annotation for watch rejections due to storage initialization: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ShazaAldawamneh
Once this PR has been reviewed and has the lgtm label, please assign benluddy for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• DOWNSTREAM_OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@ShazaAldawamneh: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

• 188e34f|Add audit annotation for watch rejections due to storage initialization: does not specify an upstream backport in the commit message
• 9c8b27a|Add audit annotation for watch rejections due to storage initialization: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[openshift-ci-robot]

@ShazaAldawamneh: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

• 1e6e5f2|UPSTREAM: : Add audit annotation for watch rejections due to storage initialization: does not specify an upstream backport in the commit message
• 32790a0|UPSTREAM: : Add audit annotation for watch rejections due to storage initialization: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[p0lyn0mial]

this looks like something that could potentially be added upstream. Have we tried it before and it was rejected?

[ShazaAldawamneh]

No it hasn't been rejected before, the main reason I’m working on this change is to get it implemented upstream, as @benluddy suggested. Currently, it’s implemented downstream in OpenShift to add audit annotations for watch rejections due to storage initialisation.

Once the downstream implementation is stable and reviewed, the plan is to carry it upstream. That’s why the annotations currently use the openshift.io namespace.

[openshift-ci-robot]

@ShazaAldawamneh: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

• 1e6e5f2|UPSTREAM: : Add audit annotation for watch rejections due to storage initialization: does not specify an upstream backport in the commit message
• 32790a0|UPSTREAM: : Add audit annotation for watch rejections due to storage initialization: does not specify an upstream backport in the commit message
• 9fbc4ac|UPSTREAM: : Add audit annotations for watch rejections due to storage initializing: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[openshift-ci]

@ShazaAldawamneh: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-scos-e2e-aws-ovn	32790a0	link	false	/test okd-scos-e2e-aws-ovn
ci/prow/e2e-aws-ovn-techpreview	9fbc4ac	link	false	/test e2e-aws-ovn-techpreview
ci/prow/k8s-e2e-conformance-aws	9fbc4ac	link	true	/test k8s-e2e-conformance-aws
ci/prow/e2e-aws-ovn-techpreview-serial	9fbc4ac	link	false	/test e2e-aws-ovn-techpreview-serial

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.