feat(events): add webhook event listener with HMAC signing and retry

[vigneshio]

Related to #5082.

Polaris has push-style event listeners for OpenTelemetry (needs an OTLP collector) and AWS CloudWatch (AWS-only), but nothing for generic HTTP endpoints. Many SIEM/audit systems (Splunk HEC, Elastic, Opensearch, Datadog, custom collectors) accept plain HTTP+JSON webhooks.

This adds a webhook event listener (polaris.event-listener.types=webhook) that:

• POSTs each sanitized event as JSON to polaris.event-listener.webhook.endpoint.
• Optionally signs payloads with HMAC-SHA256 (polaris.event-listener.webhook.secret), sent as X-Polaris-Signature-256: sha256=<hex> so receivers can verify authenticity (GitHub-webhook style). An X-Polaris-Event header carries the event type.
• Supports custom headers (polaris.event-listener.webhook.headers), e.g. Authorization for receivers that require it (Splunk HEC, etc.).
• Retries failed deliveries (non-2xx or connection errors) with exponential backoff (max-attempts, retry-backoff), then drops the event and logs an error.
• Receives only sanitized events (no RawEventAccess), so denylisted attributes such as credentials are never delivered.

Delivery is best-effort (in-memory retries, lost on restart) - same semantics as the CloudWatch listener; this is documented in the configuration reference, which also documents the payload schema and signature verification.

Implementation notes: packaged as an extension module under extensions/events/webhook, following the layout of the Kafka event listener (#4923). Listener code is modeled on AwsCloudWatchEventListener; uses the JDK java.net.http.HttpClient and javax.crypto only, so no new dependencies.

Checklist

• Don't disclose security issues! (contact security@apache.org)
• Clearly explained why the changes are needed, or linked related issues: Related to Add a generic webhook event listener for audit/SIEM integration #5082
• Added/updated tests with good coverage, or manually tested (and explained how)
• Added comments for complex logic
• Updated CHANGELOG.md (if needed)
• Updated documentation in site/content/in-dev/unreleased (if needed)

[snazy]

Thanks for the contribution, @vigneshio!

I like the direction here.
A direct webhook listener would be a useful low-infrastructure option for getting Polaris events into SIEM and audit systems.

My main concern is not the webhook idea itself.
It is that this PR introduces another external JSON representation, alongside CloudWatch and the Kafka work in #4923, while also taking on some non-trivial delivery semantics.

A few areas I think we should address or explicitly agree on:

• Shared event envelope:
  Ideally CloudWatch, Kafka, and webhooks should use one versioned Polaris event representation.
  The webhook payload currently omits the event ID and uses delivery time rather than event.metadata().timestamp().
  Including a stable event ID, original event time, and schema version would let receivers identify retry duplicates and make future evolution safer.
  A CloudEvents-shaped envelope may be worth considering, although it would not necessarily require adding its SDK.
• Bounded delivery:
  onEvent() starts an asynchronous HTTP request and returns immediately.
  This allows the Polaris listener backlog to drain while HTTP requests and scheduled retries continue accumulating independently.
  A slow endpoint could therefore create unbounded in-flight work, and events may arrive out of order.
  I think the listener needs an explicit bounded queue/concurrency model and documented ordering semantics.
• Delivery guarantees:
  The documentation currently suggests enabling persistence-in-memory-buffer for stronger delivery guarantees.
  That listener preserves a separate EventEntity record, but it is not a webhook spool and cannot replay a delivery lost during restart or after retries are exhausted.
  I would reword this unless there is an actual replay path.
• Retry policy:
  I would retry only failures classified as transient, respect Retry-After where applicable, and use capped backoff with jitter.
  This should operate within the same concurrency bound rather than creating an independent unbounded retry workload.
• Transport and deployment safety:
  HMAC protects authenticity and integrity, but not confidentiality.
  Since custom authentication headers and audit data can currently be sent over plain HTTP, I would add URI-scheme validation and a production-readiness check for non-HTTPS endpoints.
  Reserved headers should also not be overridable through the custom-header map.
  Custom trust stores and mTLS will likely be needed by some deployments, though those could potentially be follow-up work.
• Observability:
  Delivery success/failure, retries, dropped events, queue depth, in-flight requests, and latency should be measurable rather than available only through logs.

For deployments that require durable delivery, I suspect a Kafka-consuming webhook processor is the cleaner architecture:
Kafka provides buffering and replay, while the processor owns endpoint-specific retry, TLS, authentication, rate limiting, and dead-letter handling.
I do not think that invalidates a direct webhook listener, though.
A bounded and explicitly best-effort direct listener would still be valuable as the simpler option.

[vigneshio]

Thanks for the contribution, @vigneshio!

I like the direction here. A direct webhook listener would be a useful low-infrastructure option for getting Polaris events into SIEM and audit systems.

My main concern is not the webhook idea itself. It is that this PR introduces another external JSON representation, alongside CloudWatch and the Kafka work in #4923, while also taking on some non-trivial delivery semantics.

A few areas I think we should address or explicitly agree on:

• Shared event envelope:
  Ideally CloudWatch, Kafka, and webhooks should use one versioned Polaris event representation.
  The webhook payload currently omits the event ID and uses delivery time rather than event.metadata().timestamp().
  Including a stable event ID, original event time, and schema version would let receivers identify retry duplicates and make future evolution safer.
  A CloudEvents-shaped envelope may be worth considering, although it would not necessarily require adding its SDK.
• Bounded delivery:
  onEvent() starts an asynchronous HTTP request and returns immediately.
  This allows the Polaris listener backlog to drain while HTTP requests and scheduled retries continue accumulating independently.
  A slow endpoint could therefore create unbounded in-flight work, and events may arrive out of order.
  I think the listener needs an explicit bounded queue/concurrency model and documented ordering semantics.
• Delivery guarantees:
  The documentation currently suggests enabling persistence-in-memory-buffer for stronger delivery guarantees.
  That listener preserves a separate EventEntity record, but it is not a webhook spool and cannot replay a delivery lost during restart or after retries are exhausted.
  I would reword this unless there is an actual replay path.
• Retry policy:
  I would retry only failures classified as transient, respect Retry-After where applicable, and use capped backoff with jitter.
  This should operate within the same concurrency bound rather than creating an independent unbounded retry workload.
• Transport and deployment safety:
  HMAC protects authenticity and integrity, but not confidentiality.
  Since custom authentication headers and audit data can currently be sent over plain HTTP, I would add URI-scheme validation and a production-readiness check for non-HTTPS endpoints.
  Reserved headers should also not be overridable through the custom-header map.
  Custom trust stores and mTLS will likely be needed by some deployments, though those could potentially be follow-up work.
• Observability:
  Delivery success/failure, retries, dropped events, queue depth, in-flight requests, and latency should be measurable rather than available only through logs.

For deployments that require durable delivery, I suspect a Kafka-consuming webhook processor is the cleaner architecture: Kafka provides buffering and replay, while the processor owns endpoint-specific retry, TLS, authentication, rate limiting, and dead-letter handling. I do not think that invalidates a direct webhook listener, though. A bounded and explicitly best-effort direct listener would still be valuable as the simpler option.

Thanks @snazy , this was super helpful - PTAL..

• Listener is now bounded and best-effort, not an open-ended delivery pipe. payload is CloudEvents-shaped without pulling in the SDK: specversion/id/type/source/time, with time as RFC 3339 from event metadata, and extension names that actually follow the spec (realmid, deliverytime, etc). id is stable across retries so receivers can dedupe, and i kept the PolarisEventType names for type so it matches the other listeners.

• Delivery is bounded now: max-pending slots with drops + a metric when full, fixed thread pool for max-concurrent, and retries stay inside the same accounting so they can't blow up the queue. ordering is explicitly not guaranteed, and the docs say so. docs also state plainly that it's in-memory only, lost on restart, and that persistence-in-memory-buffer doesn't replay webhook deliveries - pointed anyone who needs durable delivery at a Kafka-consuming processor like you described.

• Retries are transient-only: network failures, 408, 425, 429, 5xx. anything else in 4xx drops immediately. capped backoff with full jitter, and Retry-After is honored whether it's delta-seconds or an HTTP-date.

• Require-https defaults to true and the scheme is validated at startup, so plain http fails fast unless the operator explicitly opts out. i read that as stronger than a readiness warning, but can add a ProductionReadinessCheck if you want. redirects are never followed, and reserved headers can't be overridden.

• Metrics cover success/failure/retries/drops, pending + in-flight gauges, and delivery latency.

• Left mTLS/trust stores, the shared serializer (blocked on Extension for publishings events to Kafka #4923 anyway), and the Kafka processor as follow-ups - can pull any of those into this PR let me knw. WDYT ??