chore(docs): reduce operational ambiguity

docs/cluster.md

@@ -207,6 +207,26 @@ Every node in a stable TrogonEventStore deployment settles into one of three rol
 
 The leader ensures that writes are persisted to its own disk, replicated to a majority of cluster nodes, and indexed on the leader so that they can be read from the leader, before acknowledging the write as successful to the client.
 
+#### Resign the current leader
+
+Use leader resignation when maintenance or load placement requires the cluster to elect a different leader
+without stopping a node. From the current leader, use the _Resign leader_ action on the Admin UI Operations page,
+or call the gRPC `event_store.client.operations.Operations/ResignNode` method as an `admin` or `ops` user.
+
+The gRPC response confirms that the request was accepted, not that the election has finished. It does not
+report a later failure to reach quorum. After a majority of voting nodes acknowledges the resignation, the
+leader continues to serve reads but rejects new writes and persistent-subscription requests, lets requests
+already in progress drain, and starts a new election. Monitor the node roles and client operations until the
+cluster has one leader again.
+
+Send the request to the current leader. A request sent to another node is accepted by the gRPC transport but is
+ignored by the election service. Applications must tolerate temporary `NotReady` responses while the election
+completes and follow the discovery and retry guidance for their client.
+
+Resignation makes the previous leader less preferable than another equally up-to-date candidate. It does not
+exclude that node from the election, so the same node can become leader again when it remains the best eligible
+candidate.
+
 ### Follower
 
 A cluster assigns the follower role based on an election process. A cluster uses one or more nodes with the follower role to form the quorum, or the majority of nodes necessary to confirm that the write is persisted.

docs/diagnostics/README.md

@@ -4,6 +4,7 @@ TrogonEventStore provides several ways to diagnose and troubleshoot issues.
 
 - [Logging](logs.md): structured or plain-text logs on the console and in log files.
 - [Metrics](metrics.md): collect standard metrics using Prometheus or OpenTelemetry.
+- [Monitoring and alerting](monitoring.md): turn health, metrics, and logs into operational signals.
 - [Stats](#statistics): runtime statistics exposed through the monitoring gRPC service.
 
 You can also use external tools to measure the performance of TrogonEventStore and monitor the cluster health. Learn more on the [Integrations](./integrations.md) page.

docs/diagnostics/monitoring.md

@@ -0,0 +1,107 @@
+---
+title: "Monitoring and alerting"
+---
+
+# Monitoring and alerting
+
+Monitor every node independently and evaluate cluster-wide conditions from the combined signals. The
+[metrics reference](metrics.md) defines every metric, while [monitoring integrations](integrations.md) explains
+how to collect metrics, logs, and traces with Prometheus or OpenTelemetry.
+
+The HTTP health and metrics endpoints are available without application authentication. Restrict access at the
+network or platform boundary when the telemetry must not be exposed outside the monitoring network.
+
+## Start with availability
+
+Use the HTTP health endpoints for platform probes:
+
+- `/-/liveness` reports whether the process is alive. Use it to detect a process that must be restarted.
+- `/-/readiness` reports whether the node has completed startup and is not shutting down. Remove an unready node
+  from client and load-balancer traffic without treating normal startup or graceful shutdown as a crashed
+  process. Readiness does not report leader availability during an election.
+
+Alert when a node remains unready beyond the duration established by normal startup and planned
+maintenance. For a cluster, also use `eventstore_statuses{name="Node",status=...}` to verify that the expected
+number of voting nodes is available and that exactly one node is the leader. The current status is the status
+label on the newest sample for each node. Do not count stale series for statuses the node previously held.
+
+## Alert on symptoms and trends
+
+Begin with workload-specific baselines. Alert on sustained deviations rather than isolated samples, then tune
+the evaluation window to remain longer than the configured scrape interval.
+
+### Repeated elections
+
+Alert on an unexpected increase in `eventstore_elections_count`. Frequent leadership changes interrupt traffic
+and usually indicate node, network, or resource instability.
+
+### Replication falling behind
+
+Compare `eventstore_checkpoints{name="writer"}` on the leader with the same writer checkpoint on each follower.
+A follower whose writer checkpoint does not converge reduces the cluster's recovery margin. A restored node can
+lag temporarily, but the difference should trend toward zero.
+
+### Saturated processing queues
+
+Alert on sustained growth in `eventstore_queue_length_items` or
+`eventstore_queue_queueing_duration_max_seconds`, and on a high rate of change in the cumulative
+`eventstore_queue_busy_seconds` counter. Growing queues show that work arrives faster than the node can process
+it. Correlate the affected queue group with CPU, storage latency, and request load.
+
+### gRPC failures or deadlines
+
+Alert on increases in `eventstore_incoming_grpc_calls{kind="failed"}` or
+`eventstore_incoming_grpc_calls{kind="deadline-exceeded"}`, and on failed
+`eventstore_grpc_method_duration_seconds` observations. Break failures down by operation and correlate them with
+elections, queue pressure, and network errors.
+
+### Persistent subscription lag
+
+Alert on a growing difference between the last-known and checkpointed event position for a subscription. A
+growing gap means consumers are not keeping up. Also alert on parked messages and the age of the oldest parked
+message.
+
+### Projection failure
+
+Alert when `eventstore_projection_status{status="Faulted"}` is `1`, when a required projection stops
+unexpectedly, or when its progress declines. A required system projection that stops advancing can leave its
+generated streams stale.
+
+### Resource exhaustion
+
+Alert on low free disk or memory, sustained CPU pressure, growing thread-pool pending work, or long garbage
+collection pauses. Resource pressure can increase queue time, trigger timeouts, and make a node miss cluster
+heartbeats.
+
+The persistent-subscription position metrics differ by source. Stream subscriptions expose
+`eventstore_persistent_sub_last_known_event_number` and
+`eventstore_persistent_sub_checkpointed_event_number`. Subscriptions to `$all` expose the corresponding
+`eventstore_persistent_sub_last_known_event_commit_position` and
+`eventstore_persistent_sub_checkpointed_event_commit_position` metrics.
+
+## Keep logs in the same incident view
+
+Metrics show that behavior changed; structured logs usually explain why. Collect logs from every node and
+correlate them by timestamp with health changes and metric alerts. Prioritize alerts for:
+
+- repeated election, gossip, replication, or certificate-validation errors;
+- a node that cannot open or verify its database or index;
+- a projection entering the faulted state;
+- persistent-subscription messages parked because retries were exhausted;
+- an unexpected process shutdown or startup failure.
+
+Do not alert on every warning in isolation. Establish the warnings expected during maintenance, such as a
+rolling certificate update, and alert when they persist beyond that operation's planned window.
+
+## Validate the monitoring path
+
+Before relying on an alert in production:
+
+1. Confirm every node is represented in the telemetry backend.
+2. Gracefully stop a test node and verify that readiness fails without a liveness-triggered forced restart.
+3. Perform a controlled leader resignation and verify the election, node-role, and client-retry signals.
+4. Pause a test subscription consumer and verify that its checkpoint gap grows. Separately, reject test
+   messages until they are parked and verify the parked-message signals.
+5. Test notification routing and record the dashboard, logs, and runbook needed to investigate each alert.
+
+Repeat these checks after changing metric configuration, scrape intervals, exporters, or alert labels.

docs/installation.md

@@ -91,6 +91,19 @@ source build does not register itself automatically.
 
 The node handles Service Control Manager stop requests through graceful shutdown.
 
+::: warning Windows startup deadline
+Windows Service Control Manager expects a service to report that it is running within its configured startup
+deadline. Configuration, certificate loading, and host construction occur before TrogonEventStore connects to
+the service manager, so delays or failures in that work can exhaust the deadline. Windows records service
+startup timeouts as Service Control Manager events such as 7000 or 7011.
+
+Service Control Manager can report the service as running before the database is ready to accept traffic.
+Always use `/-/readiness` as the traffic gate. Before registering the node as a service, start the same command
+interactively to expose configuration or certificate delays. If Windows stops an otherwise healthy startup,
+investigate the delay first, then adjust the host's `ServicesPipeTimeout` policy if it is expected. Microsoft
+documents that policy in [A slow service does not start due to time-out error in Windows](https://learn.microsoft.com/en-us/troubleshoot/windows-server/system-management-components/service-not-start-events-7000-7011-time-out-error).
+:::
+
 Example service registration:
 
 ```powershell:no-line-numbers

docs/projections.md

@@ -188,8 +188,8 @@ You cannot configure this projection.
 
 ### By correlation ID
 
-The `$by_correlation_id` projection links existing
-events from projections to a new stream with a stream id in the format `$bc-<correlation id>`.
+The `$by_correlation_id` projection links existing events whose JSON metadata contains the configured
+correlation property to a new stream with a stream id in the format `$bc-<correlation id>`.
 
 The projection takes one parameter, a JSON string as a projection source:
 

docs/sidebar.js

@@ -45,6 +45,7 @@ module.exports = [
             "README.md",
             "logs.md",
             "metrics.md",
+            "monitoring.md",
             "integrations.md",
         ]
     }

docs/streams.md

@@ -124,6 +124,35 @@ If you plan to use projections and delete streams, there are some considerations
 
 ## System events and streams
 
+Stream and event names beginning with `$` are reserved for database-managed data. The metadata stream for a
+system stream consequently begins with `$$$`, because stream metadata adds `$$` to the original stream name.
+Unless a stream-specific ACL overrides it, system streams use the `$systemStreamAcl` from `$settings`, which
+restricts reads and writes to `$admins` by default. See [Default ACL](security.md#default-acl) before granting
+broader access, because changing the system default also affects sensitive streams such as `$settings`.
+
+Do not append application events directly to streams owned by the projection subsystem. Projections use their
+own output and checkpoint history to recover deterministically, and an unexpected event can fault a projection.
+
+### Streams created by system projections
+
+Enabled [system projections](projections.md#system-projections) create the following reserved streams:
+
+| Projection              | Stream                                      | Purpose                                      |
+|:------------------------|:--------------------------------------------|:---------------------------------------------|
+| `$streams`              | `$streams`                                  | A link to the first event in each source stream |
+| `$by_category`          | `$ce-{category}`                            | Links to events grouped by stream category   |
+| `$by_event_type`        | `$et-{event-type}`                          | Links to events grouped by event type        |
+| `$by_event_type`        | `$et`                                       | Event-type index checkpoints                 |
+| `$by_correlation_id`    | `$bc-{correlation-id}`                      | Links grouped by metadata correlation ID     |
+| `$stream_by_category`   | `$category-{category}`                      | References to streams grouped by category    |
+
+Deleting an original event does not delete an existing link event from a system-projection stream. Consumers
+must tolerate links that no longer resolve after deletion and scavenging.
+
+Under the built-in ACL authorization policy, the projection runtime gives `$all` read and metadata-read access
+to its generated output streams. Write and delete access remains protected by the system-stream ACL. Review any
+custom stream ACL before depending on this access from an application.
+
 ### **`$persistentSubscriptionConfig`**
 
 **`$persistentSubscriptionConfig`** is a specialized paged stream that stores all configuration events for all