Rust Server train: system metrics, Prometheus, payload cleanup, repair pass, leases, limits, standalone activities, deployments, routing (#675)

Rust Server train: definition routing, worker admission, poll waits, schedule probe

Author: rmcdaniel

.github/workflows/server-parity.yml

@@ -0,0 +1,65 @@
+# October 22, workflow definition routing
+
+- **Gap.** Native records the definition fingerprint a start's worker cohort
+  agrees on, with `workflow_definition_fingerprint_source: worker`, as PHP
+  does. It then offered that run's workflow tasks to any worker, so a worker
+  with another definition could take a pinned run while the recorded one was
+  live.
+- **Source.** Server 2.10.0's `WorkflowDefinitionRouting::forWorker`,
+  which `WorkflowTaskPoller::matchesRecordedWorkflowDefinition` applies at
+  claim, and `WorkflowReplayEligibility`.
+  - **Decision.** For a workflow task of a worker-pinned run:
+    - a worker advertising the recorded fingerprint for the type takes it
+      (`matching_definition`);
+    - without a recorded fingerprint (`recorded_definition_missing`), or
+      without one from the worker (`worker_definition_missing`), nobody
+      does;
+    - a worker with another definition waits while an eligible peer
+      advertises the recorded one (`matching_definition_preferred`);
+    - otherwise it takes the task only if it advertises
+      `workflow_entry_arguments_validation` (`replay_required`, else
+      `entry_validation_unavailable`).
+
+    An unpinned run goes to any worker (`not_worker_pinned`). Native
+    refuses to start with `DW_V2_PIN_TO_RECORDED_FINGERPRINT` off, so
+    pinning is always on.
+  - **Eligible peers** (`eligibleWorkers`). Active registrations on the
+    run's namespace and queue that:
+    - match the build the task or run requires;
+    - support the type, with workflow capacity above zero;
+    - replay the run (`WorkflowReplayEligibility::allows`). That means a
+      heartbeat inside the stale window and cooperative-cancellation support
+      when the run has a request, at the protocol version the worker last
+      reported. It also means the memo and typed search attribute
+      capabilities its history needs.
+  - **Protocol version.** Each registration and heartbeat records its
+    protocol version under `_worker_protocol_version` in the worker's process
+    metrics, after the metrics it reported. A heartbeat without metrics
+    keeps the stored ones. Worker listings therefore show the key, as PHP
+    2.10.0's do; 2.5.14 had no such key.
+- **Reports** (`inspect` and `forQueue`).
+  - Describe, repair and debug report `definition_routing` for the run's
+    next due ready task:
+    - the context (`pinning_enabled`, `worker_pinned`,
+      `recorded_fingerprint`, type, queue and required build);
+    - PHP's status, the counts of eligible workers by definition, up to ten
+      example workers, `blocked` and `runnable_task`;
+    - PHP's resolution text.
+  - A blocked route adds debug's `workflow_definition_routing_blocked`
+    warning and `diagnostic_status: needs_attention`.
+  - Task queue describe samples up to ten runs with a due ready task. A
+    blocked sample sets `repair.needs_attention`.
+- **Tests** (`definition_routing.rs`).
+  - A pinned run goes to the worker with its recorded definition. Workers
+    with another definition, or without one, wait even when they validate
+    entry arguments.
+  - Once the recorded definition leaves, only an entry-validating worker
+    takes the run.
+  - An unpinned run goes to any worker.
+  - Registrations and heartbeats record the protocol version with the
+    reported metrics.
+  - Describe, debug, task queue and repair report a blocked route, then the
+    route an entry-validating replacement unblocks.
+  - The worker listing test now expects the protocol key.
+  - Full SQLite suite: 398 passed. Routing, worker, poll, child, cooperative,
+    debug, task queue, repair and describe tests: PostgreSQL 134, MySQL 134.

docs/rust-server-port-checkpoints/2026-10-22-definition-routing.md

@@ -0,0 +1,42 @@
+# October 22, worker poll waits and operation names
+
+- **Gap.** A workflow or activity poll without `timeout_seconds` waited 5
+  seconds; PHP waits `DW_WORKER_POLL_TIMEOUT` seconds, 30 by default. A
+  query-task poll without one also waited 5 seconds where PHP starts from
+  the same setting. Native refused an explicit `null` wait, which PHP's
+  `nullable` rule accepts.
+- **Source.** Server 2.10.0's `LongPoller::until`,
+  `WorkflowQueryTaskBroker::queryPollTimeoutSeconds` and
+  `WorkflowController::validateOperationName`.
+- **Poll waits.**
+  - A workflow or activity poll without a wait, or with `null`, waits
+    `max(0, DW_WORKER_POLL_TIMEOUT)` seconds. The setting is not capped.
+  - A query-task poll waits the requested time or that setting, capped by
+    `DW_QUERY_TASK_POLL_TIMEOUT` (default 5); zero answers at once.
+  - A requested wait passes Laravel's `integer` rule (so `"5"` is five
+    seconds) and must lie from 0 to 60.
+- **Operation names.**
+  - Signal, update and query requests check the name first, after the
+    payload transport check, as PHP's controller does. Empty names, names
+    over 256 bytes and names with ASCII control characters get PHP's
+    `validation_failed` errors and messages.
+  - PHP's signal and update name columns hold 255 bytes. Native keeps its
+    storage guard and refuses a 256-byte or all-space name with
+    `invalid_signal_name` or `invalid_update_name`. Query names are not
+    stored in those columns, so a 256-byte query name is accepted.
+- **Fixed along the way.** A query poll from a worker registered without
+  `capabilities` answered 503 `storage_unavailable`, because the claim read
+  the missing list as required. It now polls empty, as PHP's `stringArray`
+  does.
+- **Tests** (`poll_waits.rs`).
+  - Workflow, activity and query polls without a wait, or with `null`, take
+    the configured second.
+  - A zero setting and a requested `"0"` answer at once.
+  - Requested waits of 61, -1, `"soon"` and 1.5 are refused.
+  - A task started during a default wait is claimed.
+  - A 255-byte signal name is accepted; 256-byte signal and update names
+    are refused with the storage reasons. Too-long names and names with
+    control characters get PHP's messages for each kind.
+  - Full SQLite run (all targets): 228 unit, 397 execution and 2 upgrade
+    tests passed. Poll, signal, update and query tests: PostgreSQL 122,
+    MySQL 122.

docs/rust-server-port-checkpoints/2026-10-22-poll-waits.md

@@ -0,0 +1,60 @@
+# October 22, worker registration admission
+
+- **Gap.** Native accepted a registration without the `capability_manifest`
+  PHP requires from protocol 1.18. It also skipped most of PHP's request
+  rules and capability guards, refused a missing `worker_id` or type list,
+  and answered metadata-capability mismatches in its own shape.
+- **Source.** Server 2.10.0 `WorkerController::register`:
+  - its Laravel rules;
+  - the prepared-local-activity, Nexus and cooperative-cancellation guards;
+  - `guardPortableWorkerCapabilityManifest`;
+  - `WorkflowMetadataCapabilityPolicy::firstProtocolMismatch`;
+  - the message-streams floor;
+  - `portableWorkerCapabilityManifest`.
+- **Request rules.** Laravel trims strings and turns empty ones into null
+  first; native now does the same for the whole body.
+  - Every field follows PHP's rules: `runtime` from PHP's list, sizes,
+    capacities, task slots, process metrics, heartbeat interval.
+  - `capability_manifest` is required from protocol 1.18 and must hold the
+    `local_activities`, `worker_sessions` and `sticky_execution` entries.
+    Each entry needs `supported` (boolean) and `minimum_protocol_version`.
+  - Failures answer Laravel's 422 `validation_failed` with its messages. The
+    rule engine gained `numeric` and `required_array_keys`. A `*.field` rule
+    now covers only array items, as Laravel's expansion does.
+- **Guards.** Each answers 409 with `registered: false` and PHP's fields, in
+  PHP's order:
+  - `prepared_local_activity_cancellation_policy_capability_mismatch`,
+    `prepared_local_activity_group_capability_mismatch` and
+    `prepared_local_activity_capability_mismatch`;
+  - `nexus_cooperative_cancellation_capability_mismatch` and
+    `cooperative_cancellation_protocol_mismatch`, below protocol 1.20;
+  - `worker_capability_manifest_mismatch`, when a flat affinity capability
+    and its `supported` flag disagree;
+  - `worker_capability_protocol_floor_mismatch`, when an entry's floor is
+    below 1.18 or above the request's protocol;
+  - `workflow_metadata_capability_protocol_mismatch`, for `memo_upserts`
+    below 1.14 and `typed_search_attributes` below 1.16. This now uses PHP's
+    `capability`, `requested_version` and `remediation` fields, not native's
+    former `worker_capability` and `recorded`;
+  - `message_streams_unavailable`, below 1.15.
+- **Accepted registrations.**
+  - A missing `worker_id` gets a ULID, as PHP does.
+  - Missing type lists are stored empty.
+  - Capabilities keep their non-empty strings.
+  - The manifest is stored normalized: the three entries, a trimmed floor,
+    and non-empty `implementation` and `reason`.
+- **Worker protocol bodies.** Registration refusals now carry
+  `protocol_version` and `server_capabilities`, as `WorkerProtocol::json`
+  does. Other worker-plane refusals still lack them (#46's shape sweep).
+- **Tests.**
+  - `worker_admission.rs` uses the server's own router. It covers the
+    manifest requirement and its errors, field rules with Laravel's
+    messages, the generated id, the stored manifest, and every guard.
+  - The other tests' router adds the manifest an SDK sends from 1.18,
+    matching each registration's flat capabilities.
+  - Two cooperative tests registered `cooperative_cancellation` at protocol
+    1.19, which PHP refuses. They now register at 1.20 and still poll at the
+    protocol each case tests.
+  - Full SQLite run (all targets): 230 unit, 401 execution and 2 upgrade
+    tests passed. Worker, registration, admission, cooperative and
+    namespace tests: PostgreSQL 124, MySQL 124.