From 0e182a9e398beee28b0be2b46a5b283ea69b5be8 Mon Sep 17 00:00:00 2001 From: Yordis Prieto Date: Sun, 19 Jul 2026 00:07:40 -0400 Subject: [PATCH 1/2] chore(docs): reduce operational ambiguity Signed-off-by: Yordis Prieto --- docs/cluster.md | 20 ++++++ docs/diagnostics/README.md | 1 + docs/diagnostics/monitoring.md | 107 +++++++++++++++++++++++++++++++++ docs/installation.md | 13 ++++ docs/projections.md | 4 +- docs/sidebar.js | 1 + docs/streams.md | 29 +++++++++ 7 files changed, 173 insertions(+), 2 deletions(-) create mode 100644 docs/diagnostics/monitoring.md diff --git a/docs/cluster.md b/docs/cluster.md index b62008a54..9c326de56 100644 --- a/docs/cluster.md +++ b/docs/cluster.md @@ -207,6 +207,26 @@ Every node in a stable TrogonEventStore deployment settles into one of three rol The leader ensures that writes are persisted to its own disk, replicated to a majority of cluster nodes, and indexed on the leader so that they can be read from the leader, before acknowledging the write as successful to the client. +#### Resign the current leader + +Use leader resignation when maintenance or load placement requires the cluster to elect a different leader +without stopping a node. From the current leader, use the _Resign leader_ action on the Admin UI Operations page, +or call the gRPC `event_store.client.operations.Operations/ResignNode` method as an `admin` or `ops` user. + +The gRPC response confirms that the request was accepted, not that the election has finished. It does not +report a later failure to reach quorum. After a majority of voting nodes acknowledges the resignation, the +leader continues to serve reads but rejects new writes and persistent-subscription requests, lets requests +already in progress drain, and starts a new election. Monitor the node roles and client operations until the +cluster has one leader again. + +Send the request to the current leader. A request sent to another node is accepted by the gRPC transport but is +ignored by the election service. Applications must tolerate temporary `NotReady` responses while the election +completes and follow the discovery and retry guidance for their client. + +Resignation makes the previous leader less preferable than another equally up-to-date candidate. It does not +exclude that node from the election, so the same node can become leader again when it remains the best eligible +candidate. + ### Follower A cluster assigns the follower role based on an election process. A cluster uses one or more nodes with the follower role to form the quorum, or the majority of nodes necessary to confirm that the write is persisted. diff --git a/docs/diagnostics/README.md b/docs/diagnostics/README.md index 8c5f815e7..adc52604b 100644 --- a/docs/diagnostics/README.md +++ b/docs/diagnostics/README.md @@ -4,6 +4,7 @@ TrogonEventStore provides several ways to diagnose and troubleshoot issues. - [Logging](logs.md): structured or plain-text logs on the console and in log files. - [Metrics](metrics.md): collect standard metrics using Prometheus or OpenTelemetry. +- [Monitoring and alerting](monitoring.md): turn health, metrics, and logs into operational signals. - [Stats](#statistics): runtime statistics exposed through the monitoring gRPC service. You can also use external tools to measure the performance of TrogonEventStore and monitor the cluster health. Learn more on the [Integrations](./integrations.md) page. diff --git a/docs/diagnostics/monitoring.md b/docs/diagnostics/monitoring.md new file mode 100644 index 000000000..acbc97dc9 --- /dev/null +++ b/docs/diagnostics/monitoring.md @@ -0,0 +1,107 @@ +--- +title: "Monitoring and alerting" +--- + +# Monitoring and alerting + +Monitor every node independently and evaluate cluster-wide conditions from the combined signals. The +[metrics reference](metrics.md) defines every metric, while [monitoring integrations](integrations.md) explains +how to collect metrics, logs, and traces with Prometheus or OpenTelemetry. + +The HTTP health and metrics endpoints are available without application authentication. Restrict access at the +network or platform boundary when the telemetry must not be exposed outside the monitoring network. + +## Start with availability + +Use the HTTP health endpoints for platform probes: + +- `/-/liveness` reports whether the process is alive. Use it to detect a process that must be restarted. +- `/-/readiness` reports whether the node has completed startup and is not shutting down. Remove an unready node + from client and load-balancer traffic without treating normal startup or graceful shutdown as a crashed + process. Readiness does not report leader availability during an election. + +Alert when a node remains unready beyond the duration established by normal startup and planned +maintenance. For a cluster, also use `eventstore_statuses{name="Node",status=...}` to verify that the expected +number of voting nodes is available and that exactly one node is the leader. The current status is the status +label on the newest sample for each node. Do not count stale series for statuses the node previously held. + +## Alert on symptoms and trends + +Begin with workload-specific baselines. Alert on sustained deviations rather than isolated samples, then tune +the evaluation window to remain longer than the configured scrape interval. + +### Repeated elections + +Alert on an unexpected increase in `eventstore_elections_count`. Frequent leadership changes interrupt traffic +and usually indicate node, network, or resource instability. + +### Replication falling behind + +Compare `eventstore_checkpoints{name="writer"}` on the leader with the same writer checkpoint on each follower. +A follower whose writer checkpoint does not converge reduces the cluster's recovery margin. A restored node can +lag temporarily, but the difference should trend toward zero. + +### Saturated processing queues + +Alert on sustained growth in `eventstore_queue_length_items` or +`eventstore_queue_queueing_duration_max_seconds`, and on a high rate of change in the cumulative +`eventstore_queue_busy_seconds` counter. Growing queues show that work arrives faster than the node can process +it. Correlate the affected queue group with CPU, storage latency, and request load. + +### gRPC failures or deadlines + +Alert on increases in `eventstore_incoming_grpc_calls{kind="failed"}` or +`eventstore_incoming_grpc_calls{kind="deadline-exceeded"}`, and on failed +`eventstore_grpc_method_duration_seconds` observations. Break failures down by operation and correlate them with +elections, queue pressure, and network errors. + +### Persistent subscription lag + +Alert on a growing difference between the last-known and checkpointed event position for a subscription. A +growing gap means consumers are not keeping up. Also alert on parked messages and the age of the oldest parked +message. + +### Projection failure + +Alert when `eventstore_projection_status{status="Faulted"}` is `1`, when a required projection stops +unexpectedly, or when its progress declines. A required system projection that stops advancing can leave its +generated streams stale. + +### Resource exhaustion + +Alert on low free disk or memory, sustained CPU pressure, growing thread-pool pending work, or long garbage +collection pauses. Resource pressure can increase queue time, trigger timeouts, and make a node miss cluster +heartbeats. + +The persistent-subscription position metrics differ by source. Stream subscriptions expose +`eventstore_persistent_sub_last_known_event_number` and +`eventstore_persistent_sub_checkpointed_event_number`. Subscriptions to `$all` expose the corresponding +`eventstore_persistent_sub_last_known_event_commit_position` and +`eventstore_persistent_sub_checkpointed_event_commit_position` metrics. + +## Keep logs in the same incident view + +Metrics show that behavior changed; structured logs usually explain why. Collect logs from every node and +correlate them by timestamp with health changes and metric alerts. Prioritize alerts for: + +- repeated election, gossip, replication, or certificate-validation errors; +- a node that cannot open or verify its database or index; +- a projection entering the faulted state; +- persistent-subscription messages parked because retries were exhausted; +- an unexpected process shutdown or startup failure. + +Do not alert on every warning in isolation. Establish the warnings expected during maintenance, such as a +rolling certificate update, and alert when they persist beyond that operation's planned window. + +## Validate the monitoring path + +Before relying on an alert in production: + +1. Confirm every node is represented in the telemetry backend. +2. Gracefully stop a test node and verify that readiness fails without a liveness-triggered forced restart. +3. Perform a controlled leader resignation and verify the election, node-role, and client-retry signals. +4. Pause a test subscription consumer and verify that its checkpoint gap grows. Separately, reject test + messages until they are parked and verify the parked-message signals. +5. Test notification routing and record the dashboard, logs, and runbook needed to investigate each alert. + +Repeat these checks after changing metric configuration, scrape intervals, exporters, or alert labels. diff --git a/docs/installation.md b/docs/installation.md index 3a76d2668..07f44ba23 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -91,6 +91,19 @@ source build does not register itself automatically. The node handles Service Control Manager stop requests through graceful shutdown. +::: warning Windows startup deadline +Windows Service Control Manager expects a service to report that it is running within its configured startup +deadline. Configuration, certificate loading, and host construction occur before TrogonEventStore connects to +the service manager, so delays or failures in that work can exhaust the deadline. Windows records service +startup timeouts as Service Control Manager events such as 7000 or 7011. + +Service Control Manager can report the service as running before the database is ready to accept traffic. +Always use `/-/readiness` as the traffic gate. Before registering the node as a service, start the same command +interactively to expose configuration or certificate delays. If Windows stops an otherwise healthy startup, +investigate the delay first, then adjust the host's `ServicesPipeTimeout` policy if it is expected. Microsoft +documents that policy in [A slow service does not start due to time-out error in Windows](https://learn.microsoft.com/en-us/troubleshoot/windows-server/system-management-components/service-not-start-events-7000-7011-time-out-error). +::: + Example service registration: ```powershell:no-line-numbers diff --git a/docs/projections.md b/docs/projections.md index ffe49724d..f4afb59cb 100644 --- a/docs/projections.md +++ b/docs/projections.md @@ -188,8 +188,8 @@ You cannot configure this projection. ### By correlation ID -The `$by_correlation_id` projection links existing -events from projections to a new stream with a stream id in the format `$bc-`. +The `$by_correlation_id` projection links existing events whose JSON metadata contains the configured +correlation property to a new stream with a stream id in the format `$bc-`. The projection takes one parameter, a JSON string as a projection source: diff --git a/docs/sidebar.js b/docs/sidebar.js index 5e3bec365..681f4e52b 100644 --- a/docs/sidebar.js +++ b/docs/sidebar.js @@ -45,6 +45,7 @@ module.exports = [ "README.md", "logs.md", "metrics.md", + "monitoring.md", "integrations.md", ] } diff --git a/docs/streams.md b/docs/streams.md index df9885503..dd84749c2 100644 --- a/docs/streams.md +++ b/docs/streams.md @@ -124,6 +124,35 @@ If you plan to use projections and delete streams, there are some considerations ## System events and streams +Stream and event names beginning with `$` are reserved for database-managed data. The metadata stream for a +system stream consequently begins with `$$$`, because stream metadata adds `$$` to the original stream name. +Unless a stream-specific ACL overrides it, system streams use the `$systemStreamAcl` from `$settings`, which +restricts reads and writes to `$admins` by default. See [Default ACL](security.md#default-acl) before granting +broader access, because changing the system default also affects sensitive streams such as `$settings`. + +Do not append application events directly to streams owned by the projection subsystem. Projections use their +own output and checkpoint history to recover deterministically, and an unexpected event can fault a projection. + +### Streams created by system projections + +Enabled [system projections](projections.md#system-projections) create the following reserved streams: + +| Projection | Stream | Purpose | +|:------------------------|:--------------------------------------------|:---------------------------------------------| +| `$streams` | `$streams` | A link to the first event in each source stream | +| `$by_category` | `$ce-{category}` | Links to events grouped by stream category | +| `$by_event_type` | `$et-{event-type}` | Links to events grouped by event type | +| `$by_event_type` | `$et` | Event-type index checkpoints | +| `$by_correlation_id` | `$bc-{correlation-id}` | Links grouped by metadata correlation ID | +| `$stream_by_category` | `$category-{category}` | References to streams grouped by category | + +Deleting an original event does not delete an existing link event from a system-projection stream. Consumers +must tolerate links that no longer resolve after deletion and scavenging. + +Under the built-in ACL authorization policy, the projection runtime gives `$all` read and metadata-read access +to its generated output streams. Write and delete access remains protected by the system-stream ACL. Review any +custom stream ACL before depending on this access from an application. + ### **`$persistentSubscriptionConfig`** **`$persistentSubscriptionConfig`** is a specialized paged stream that stores all configuration events for all From 06a75611ee4479549a9a5235631758f085770599 Mon Sep 17 00:00:00 2001 From: Yordis Prieto Date: Sun, 19 Jul 2026 00:12:44 -0400 Subject: [PATCH 2/2] chore(docs): make recovery guidance actionable Signed-off-by: Yordis Prieto --- docs/diagnostics/monitoring.md | 12 ++++++------ docs/installation.md | 6 ++++-- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/diagnostics/monitoring.md b/docs/diagnostics/monitoring.md index acbc97dc9..34f7c8dda 100644 --- a/docs/diagnostics/monitoring.md +++ b/docs/diagnostics/monitoring.md @@ -61,6 +61,12 @@ Alert on a growing difference between the last-known and checkpointed event posi growing gap means consumers are not keeping up. Also alert on parked messages and the age of the oldest parked message. +The persistent-subscription position metrics differ by source. Stream subscriptions expose +`eventstore_persistent_sub_last_known_event_number` and +`eventstore_persistent_sub_checkpointed_event_number`. Subscriptions to `$all` expose the corresponding +`eventstore_persistent_sub_last_known_event_commit_position` and +`eventstore_persistent_sub_checkpointed_event_commit_position` metrics. + ### Projection failure Alert when `eventstore_projection_status{status="Faulted"}` is `1`, when a required projection stops @@ -73,12 +79,6 @@ Alert on low free disk or memory, sustained CPU pressure, growing thread-pool pe collection pauses. Resource pressure can increase queue time, trigger timeouts, and make a node miss cluster heartbeats. -The persistent-subscription position metrics differ by source. Stream subscriptions expose -`eventstore_persistent_sub_last_known_event_number` and -`eventstore_persistent_sub_checkpointed_event_number`. Subscriptions to `$all` expose the corresponding -`eventstore_persistent_sub_last_known_event_commit_position` and -`eventstore_persistent_sub_checkpointed_event_commit_position` metrics. - ## Keep logs in the same incident view Metrics show that behavior changed; structured logs usually explain why. Collect logs from every node and diff --git a/docs/installation.md b/docs/installation.md index 07f44ba23..fb5c8561a 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -100,8 +100,10 @@ startup timeouts as Service Control Manager events such as 7000 or 7011. Service Control Manager can report the service as running before the database is ready to accept traffic. Always use `/-/readiness` as the traffic gate. Before registering the node as a service, start the same command interactively to expose configuration or certificate delays. If Windows stops an otherwise healthy startup, -investigate the delay first, then adjust the host's `ServicesPipeTimeout` policy if it is expected. Microsoft -documents that policy in [A slow service does not start due to time-out error in Windows](https://learn.microsoft.com/en-us/troubleshoot/windows-server/system-management-components/service-not-start-events-7000-7011-time-out-error). +investigate the delay first. If it is expected, increase the +`HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\ServicesPipeTimeout` `DWORD` incrementally. The value is +measured in milliseconds, and Windows must be restarted for a change to take effect. Microsoft documents this +policy in [A slow service does not start due to time-out error in Windows](https://learn.microsoft.com/en-us/troubleshoot/windows-server/system-management-components/service-not-start-events-7000-7011-time-out-error). ::: Example service registration: