diff --git a/docs/cluster.md b/docs/cluster.md index b62008a54..9c326de56 100644 --- a/docs/cluster.md +++ b/docs/cluster.md @@ -207,6 +207,26 @@ Every node in a stable TrogonEventStore deployment settles into one of three rol The leader ensures that writes are persisted to its own disk, replicated to a majority of cluster nodes, and indexed on the leader so that they can be read from the leader, before acknowledging the write as successful to the client. +#### Resign the current leader + +Use leader resignation when maintenance or load placement requires the cluster to elect a different leader +without stopping a node. From the current leader, use the _Resign leader_ action on the Admin UI Operations page, +or call the gRPC `event_store.client.operations.Operations/ResignNode` method as an `admin` or `ops` user. + +The gRPC response confirms that the request was accepted, not that the election has finished. It does not +report a later failure to reach quorum. After a majority of voting nodes acknowledges the resignation, the +leader continues to serve reads but rejects new writes and persistent-subscription requests, lets requests +already in progress drain, and starts a new election. Monitor the node roles and client operations until the +cluster has one leader again. + +Send the request to the current leader. A request sent to another node is accepted by the gRPC transport but is +ignored by the election service. Applications must tolerate temporary `NotReady` responses while the election +completes and follow the discovery and retry guidance for their client. + +Resignation makes the previous leader less preferable than another equally up-to-date candidate. It does not +exclude that node from the election, so the same node can become leader again when it remains the best eligible +candidate. + ### Follower A cluster assigns the follower role based on an election process. A cluster uses one or more nodes with the follower role to form the quorum, or the majority of nodes necessary to confirm that the write is persisted. diff --git a/docs/diagnostics/README.md b/docs/diagnostics/README.md index 8c5f815e7..adc52604b 100644 --- a/docs/diagnostics/README.md +++ b/docs/diagnostics/README.md @@ -4,6 +4,7 @@ TrogonEventStore provides several ways to diagnose and troubleshoot issues. - [Logging](logs.md): structured or plain-text logs on the console and in log files. - [Metrics](metrics.md): collect standard metrics using Prometheus or OpenTelemetry. +- [Monitoring and alerting](monitoring.md): turn health, metrics, and logs into operational signals. - [Stats](#statistics): runtime statistics exposed through the monitoring gRPC service. You can also use external tools to measure the performance of TrogonEventStore and monitor the cluster health. Learn more on the [Integrations](./integrations.md) page. diff --git a/docs/diagnostics/monitoring.md b/docs/diagnostics/monitoring.md new file mode 100644 index 000000000..34f7c8dda --- /dev/null +++ b/docs/diagnostics/monitoring.md @@ -0,0 +1,107 @@ +--- +title: "Monitoring and alerting" +--- + +# Monitoring and alerting + +Monitor every node independently and evaluate cluster-wide conditions from the combined signals. The +[metrics reference](metrics.md) defines every metric, while [monitoring integrations](integrations.md) explains +how to collect metrics, logs, and traces with Prometheus or OpenTelemetry. + +The HTTP health and metrics endpoints are available without application authentication. Restrict access at the +network or platform boundary when the telemetry must not be exposed outside the monitoring network. + +## Start with availability + +Use the HTTP health endpoints for platform probes: + +- `/-/liveness` reports whether the process is alive. Use it to detect a process that must be restarted. +- `/-/readiness` reports whether the node has completed startup and is not shutting down. Remove an unready node + from client and load-balancer traffic without treating normal startup or graceful shutdown as a crashed + process. Readiness does not report leader availability during an election. + +Alert when a node remains unready beyond the duration established by normal startup and planned +maintenance. For a cluster, also use `eventstore_statuses{name="Node",status=...}` to verify that the expected +number of voting nodes is available and that exactly one node is the leader. The current status is the status +label on the newest sample for each node. Do not count stale series for statuses the node previously held. + +## Alert on symptoms and trends + +Begin with workload-specific baselines. Alert on sustained deviations rather than isolated samples, then tune +the evaluation window to remain longer than the configured scrape interval. + +### Repeated elections + +Alert on an unexpected increase in `eventstore_elections_count`. Frequent leadership changes interrupt traffic +and usually indicate node, network, or resource instability. + +### Replication falling behind + +Compare `eventstore_checkpoints{name="writer"}` on the leader with the same writer checkpoint on each follower. +A follower whose writer checkpoint does not converge reduces the cluster's recovery margin. A restored node can +lag temporarily, but the difference should trend toward zero. + +### Saturated processing queues + +Alert on sustained growth in `eventstore_queue_length_items` or +`eventstore_queue_queueing_duration_max_seconds`, and on a high rate of change in the cumulative +`eventstore_queue_busy_seconds` counter. Growing queues show that work arrives faster than the node can process +it. Correlate the affected queue group with CPU, storage latency, and request load. + +### gRPC failures or deadlines + +Alert on increases in `eventstore_incoming_grpc_calls{kind="failed"}` or +`eventstore_incoming_grpc_calls{kind="deadline-exceeded"}`, and on failed +`eventstore_grpc_method_duration_seconds` observations. Break failures down by operation and correlate them with +elections, queue pressure, and network errors. + +### Persistent subscription lag + +Alert on a growing difference between the last-known and checkpointed event position for a subscription. A +growing gap means consumers are not keeping up. Also alert on parked messages and the age of the oldest parked +message. + +The persistent-subscription position metrics differ by source. Stream subscriptions expose +`eventstore_persistent_sub_last_known_event_number` and +`eventstore_persistent_sub_checkpointed_event_number`. Subscriptions to `$all` expose the corresponding +`eventstore_persistent_sub_last_known_event_commit_position` and +`eventstore_persistent_sub_checkpointed_event_commit_position` metrics. + +### Projection failure + +Alert when `eventstore_projection_status{status="Faulted"}` is `1`, when a required projection stops +unexpectedly, or when its progress declines. A required system projection that stops advancing can leave its +generated streams stale. + +### Resource exhaustion + +Alert on low free disk or memory, sustained CPU pressure, growing thread-pool pending work, or long garbage +collection pauses. Resource pressure can increase queue time, trigger timeouts, and make a node miss cluster +heartbeats. + +## Keep logs in the same incident view + +Metrics show that behavior changed; structured logs usually explain why. Collect logs from every node and +correlate them by timestamp with health changes and metric alerts. Prioritize alerts for: + +- repeated election, gossip, replication, or certificate-validation errors; +- a node that cannot open or verify its database or index; +- a projection entering the faulted state; +- persistent-subscription messages parked because retries were exhausted; +- an unexpected process shutdown or startup failure. + +Do not alert on every warning in isolation. Establish the warnings expected during maintenance, such as a +rolling certificate update, and alert when they persist beyond that operation's planned window. + +## Validate the monitoring path + +Before relying on an alert in production: + +1. Confirm every node is represented in the telemetry backend. +2. Gracefully stop a test node and verify that readiness fails without a liveness-triggered forced restart. +3. Perform a controlled leader resignation and verify the election, node-role, and client-retry signals. +4. Pause a test subscription consumer and verify that its checkpoint gap grows. Separately, reject test + messages until they are parked and verify the parked-message signals. +5. Test notification routing and record the dashboard, logs, and runbook needed to investigate each alert. + +Repeat these checks after changing metric configuration, scrape intervals, exporters, or alert labels. diff --git a/docs/installation.md b/docs/installation.md index 3a76d2668..fb5c8561a 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -91,6 +91,21 @@ source build does not register itself automatically. The node handles Service Control Manager stop requests through graceful shutdown. +::: warning Windows startup deadline +Windows Service Control Manager expects a service to report that it is running within its configured startup +deadline. Configuration, certificate loading, and host construction occur before TrogonEventStore connects to +the service manager, so delays or failures in that work can exhaust the deadline. Windows records service +startup timeouts as Service Control Manager events such as 7000 or 7011. + +Service Control Manager can report the service as running before the database is ready to accept traffic. +Always use `/-/readiness` as the traffic gate. Before registering the node as a service, start the same command +interactively to expose configuration or certificate delays. If Windows stops an otherwise healthy startup, +investigate the delay first. If it is expected, increase the +`HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\ServicesPipeTimeout` `DWORD` incrementally. The value is +measured in milliseconds, and Windows must be restarted for a change to take effect. Microsoft documents this +policy in [A slow service does not start due to time-out error in Windows](https://learn.microsoft.com/en-us/troubleshoot/windows-server/system-management-components/service-not-start-events-7000-7011-time-out-error). +::: + Example service registration: ```powershell:no-line-numbers diff --git a/docs/projections.md b/docs/projections.md index ffe49724d..f4afb59cb 100644 --- a/docs/projections.md +++ b/docs/projections.md @@ -188,8 +188,8 @@ You cannot configure this projection. ### By correlation ID -The `$by_correlation_id` projection links existing -events from projections to a new stream with a stream id in the format `$bc-`. +The `$by_correlation_id` projection links existing events whose JSON metadata contains the configured +correlation property to a new stream with a stream id in the format `$bc-`. The projection takes one parameter, a JSON string as a projection source: diff --git a/docs/sidebar.js b/docs/sidebar.js index 5e3bec365..681f4e52b 100644 --- a/docs/sidebar.js +++ b/docs/sidebar.js @@ -45,6 +45,7 @@ module.exports = [ "README.md", "logs.md", "metrics.md", + "monitoring.md", "integrations.md", ] } diff --git a/docs/streams.md b/docs/streams.md index df9885503..dd84749c2 100644 --- a/docs/streams.md +++ b/docs/streams.md @@ -124,6 +124,35 @@ If you plan to use projections and delete streams, there are some considerations ## System events and streams +Stream and event names beginning with `$` are reserved for database-managed data. The metadata stream for a +system stream consequently begins with `$$$`, because stream metadata adds `$$` to the original stream name. +Unless a stream-specific ACL overrides it, system streams use the `$systemStreamAcl` from `$settings`, which +restricts reads and writes to `$admins` by default. See [Default ACL](security.md#default-acl) before granting +broader access, because changing the system default also affects sensitive streams such as `$settings`. + +Do not append application events directly to streams owned by the projection subsystem. Projections use their +own output and checkpoint history to recover deterministically, and an unexpected event can fault a projection. + +### Streams created by system projections + +Enabled [system projections](projections.md#system-projections) create the following reserved streams: + +| Projection | Stream | Purpose | +|:------------------------|:--------------------------------------------|:---------------------------------------------| +| `$streams` | `$streams` | A link to the first event in each source stream | +| `$by_category` | `$ce-{category}` | Links to events grouped by stream category | +| `$by_event_type` | `$et-{event-type}` | Links to events grouped by event type | +| `$by_event_type` | `$et` | Event-type index checkpoints | +| `$by_correlation_id` | `$bc-{correlation-id}` | Links grouped by metadata correlation ID | +| `$stream_by_category` | `$category-{category}` | References to streams grouped by category | + +Deleting an original event does not delete an existing link event from a system-projection stream. Consumers +must tolerate links that no longer resolve after deletion and scavenging. + +Under the built-in ACL authorization policy, the projection runtime gives `$all` read and metadata-read access +to its generated output streams. Write and delete access remains protected by the system-stream ACL. Review any +custom stream ACL before depending on this access from an application. + ### **`$persistentSubscriptionConfig`** **`$persistentSubscriptionConfig`** is a specialized paged stream that stores all configuration events for all