Rust Server train: command batches, named namespaces, completion checks, external inputs, operator surface, leases, deployments (#696)

Rust Server (#325) oct23 train. See the PR description for its contents and evidence. Every Rust suite passed on 69485a2, and the head adds only a PHP-only main merge (#701).

Author: rmcdaniel

.github/workflows/server-parity.yml

@@ -36,6 +36,10 @@ on:
       - tests/Unit/ServerParityOperatorMetricsContractTest.mjs
       - tests/Unit/ServerParityActivityLeaseRenewalContractTest.mjs
       - tests/Unit/ServerParityServiceCatalogContractTest.mjs
+      - tests/Unit/ServerParitySystemMetricsContractTest.mjs
+      - tests/Unit/ServerParityPrometheusMetricsContractTest.mjs
+      - tests/Unit/ServerParityPayloadCleanupContractTest.mjs
+      - tests/Unit/ServerParityRepairPassContractTest.mjs
       - tests/Unit/ServerParityNamespaceDeleteContractTest.mjs
       - tests/Unit/ServerParityWaitingHistoryContractTest.mjs
       - tests/Unit/ServerParityTaskRetryContractTest.mjs
@@ -86,6 +90,10 @@ on:
       - tests/Unit/ServerParityOperatorMetricsContractTest.mjs
       - tests/Unit/ServerParityActivityLeaseRenewalContractTest.mjs
       - tests/Unit/ServerParityServiceCatalogContractTest.mjs
+      - tests/Unit/ServerParitySystemMetricsContractTest.mjs
+      - tests/Unit/ServerParityPrometheusMetricsContractTest.mjs
+      - tests/Unit/ServerParityPayloadCleanupContractTest.mjs
+      - tests/Unit/ServerParityRepairPassContractTest.mjs
       - tests/Unit/ServerParityNamespaceDeleteContractTest.mjs
       - tests/Unit/ServerParityWaitingHistoryContractTest.mjs
       - tests/Unit/ServerParityTaskRetryContractTest.mjs
@@ -443,8 +451,8 @@ jobs:
             -e DB_USERNAME=root -e DB_PASSWORD=parity-disposable-password \
             -e CACHE_STORE=file -e QUEUE_CONNECTION=database -e DW_AUTH_TOKEN=parity-test-token \
             "$layout" artisan server:bootstrap --force > parity-evidence/published-bootstrap.txt
-          # Compiling the library tests peaks near 2.5 GB; a 2 GB limit OOM-killed rustc (exit 101).
-          docker run --rm --user="$(id -u):$(id -g)" --network "$RESOURCE_SCOPE" --cpus=2 --memory=4g --memory-swap=4g \
+          # Compiling the library tests with two jobs peaks near 4.2 GB; a 4 GB limit OOM-killed rustc (exit 101).
+          docker run --rm --user="$(id -u):$(id -g)" --network "$RESOURCE_SCOPE" --cpus=2 --memory=6g --memory-swap=6g \
             -e CARGO_HOME=/cargo -e CARGO_TARGET_DIR=/target -e CARGO_BUILD_JOBS=2 \
             -e DW_TEST_MYSQL_URL=mysql://root:parity-disposable-password@parity-mysql:3306/php_ref \
             -e DW_EXECUTION_MYSQL_URL=mysql://root:parity-disposable-password@parity-mysql:3306/php_ref \
@@ -955,8 +963,8 @@ jobs:
             -e DB_USERNAME=parity -e DB_PASSWORD=parity-disposable-password \
             -e CACHE_STORE=file -e QUEUE_CONNECTION=database -e DW_AUTH_TOKEN=parity-test-token \
             "$layout" artisan server:bootstrap --force > parity-evidence/published-bootstrap.txt
-          # Compiling the library tests peaks near 2.5 GB; a 2 GB limit OOM-killed rustc (exit 101).
-          docker run --rm --user="$(id -u):$(id -g)" --network "$RESOURCE_SCOPE" --cpus=2 --memory=4g --memory-swap=4g \
+          # Compiling the library tests with two jobs peaks near 4.2 GB; a 4 GB limit OOM-killed rustc (exit 101).
+          docker run --rm --user="$(id -u):$(id -g)" --network "$RESOURCE_SCOPE" --cpus=2 --memory=6g --memory-swap=6g \
             -e CARGO_HOME=/cargo -e CARGO_TARGET_DIR=/target -e CARGO_BUILD_JOBS=2 \
             -e DW_TEST_POSTGRES_URL=postgres://parity:parity-disposable-password@parity-pg:5432/php_ref \
             -e DW_EXECUTION_POSTGRES_URL=postgres://parity:parity-disposable-password@parity-pg:5432/php_ref \
@@ -1470,8 +1478,8 @@ jobs:
           }
           trap rust_test_failures ERR
           mkdir -p cargo-home rust-target parity-evidence
-          # Compiling the library tests peaks near 2.5 GB; a 2 GB limit OOM-killed rustc (exit 101).
-          docker run --rm --user="$(id -u):$(id -g)" --cpus=2 --memory=4g --memory-swap=4g \
+          # Compiling the library tests with two jobs peaks near 4.2 GB; a 4 GB limit OOM-killed rustc (exit 101).
+          docker run --rm --user="$(id -u):$(id -g)" --cpus=2 --memory=6g --memory-swap=6g \
             -e CARGO_HOME=/cargo -e CARGO_TARGET_DIR=/target -e CARGO_BUILD_JOBS=2 \
             -e DW_UPGRADE_TEST_WRITABLE=/evidence -e DW_UPGRADE_TEST_READONLY=/source/parity-evidence \
             -v "$PWD:/source:ro" -v "$PWD/cargo-home:/cargo" -v "$PWD/rust-target:/target" \
@@ -1510,7 +1518,7 @@ jobs:
             -e COMPOSER_HOME=/tmp/composer -v "$PWD:/source" -w /source "$image" \
             install --working-dir=scripts/conformance/server-parity --no-interaction --no-progress
           docker run --rm --user=1000:1000 --entrypoint node -v "$PWD:/source:ro" -w /source \
-            "$image" --test tests/Unit/ServerParityRunnerTest.mjs tests/Unit/ServerParityCooperativeContractTest.mjs tests/Unit/ServerParityChildCancellationContractTest.mjs tests/Unit/ServerParityScheduleContractTest.mjs tests/Unit/ServerParityVisibilityContractTest.mjs tests/Unit/ServerParityAdmissionContractTest.mjs tests/Unit/ServerParityRoleTokenContractTest.mjs tests/Unit/ServerParityWorkerDeregistrationContractTest.mjs tests/Unit/ServerParitySearchAttributeContractTest.mjs tests/Unit/ServerParityWorkerListingContractTest.mjs tests/Unit/ServerParityWorkerReplacementContractTest.mjs tests/Unit/ServerParityPatchDeploymentContractTest.mjs tests/Unit/ServerParityWaitingHistoryContractTest.mjs tests/Unit/ServerParityTaskRetryContractTest.mjs tests/Unit/ServerParityPublishedPatchContractTest.mjs tests/Unit/ServerParityPublishedPatchTransportTest.mjs tests/Unit/ServerParityRecordFileTest.mjs tests/Unit/ServerParityLateWorkflowCompletionContractTest.mjs tests/Unit/ServerParityStorageTestContractTest.mjs tests/Unit/ServerParitySchedulePoliciesContractTest.mjs tests/Unit/ServerParitySignalsBeforeWorkerContractTest.mjs tests/Unit/ServerParityScheduleCatalogContractTest.mjs tests/Unit/ServerParityOperatorMetricsContractTest.mjs tests/Unit/ServerParityActivityLeaseRenewalContractTest.mjs tests/Unit/ServerParityServiceCatalogContractTest.mjs tests/Unit/ServerParityNamespaceDeleteContractTest.mjs > parity-evidence/comparator-tests.txt
+            "$image" --test tests/Unit/ServerParityRunnerTest.mjs tests/Unit/ServerParityCooperativeContractTest.mjs tests/Unit/ServerParityChildCancellationContractTest.mjs tests/Unit/ServerParityScheduleContractTest.mjs tests/Unit/ServerParityVisibilityContractTest.mjs tests/Unit/ServerParityAdmissionContractTest.mjs tests/Unit/ServerParityRoleTokenContractTest.mjs tests/Unit/ServerParityWorkerDeregistrationContractTest.mjs tests/Unit/ServerParitySearchAttributeContractTest.mjs tests/Unit/ServerParityWorkerListingContractTest.mjs tests/Unit/ServerParityWorkerReplacementContractTest.mjs tests/Unit/ServerParityPatchDeploymentContractTest.mjs tests/Unit/ServerParityWaitingHistoryContractTest.mjs tests/Unit/ServerParityTaskRetryContractTest.mjs tests/Unit/ServerParityPublishedPatchContractTest.mjs tests/Unit/ServerParityPublishedPatchTransportTest.mjs tests/Unit/ServerParityRecordFileTest.mjs tests/Unit/ServerParityLateWorkflowCompletionContractTest.mjs tests/Unit/ServerParityStorageTestContractTest.mjs tests/Unit/ServerParitySchedulePoliciesContractTest.mjs tests/Unit/ServerParitySignalsBeforeWorkerContractTest.mjs tests/Unit/ServerParityScheduleCatalogContractTest.mjs tests/Unit/ServerParityOperatorMetricsContractTest.mjs tests/Unit/ServerParityActivityLeaseRenewalContractTest.mjs tests/Unit/ServerParityServiceCatalogContractTest.mjs tests/Unit/ServerParitySystemMetricsContractTest.mjs tests/Unit/ServerParityPrometheusMetricsContractTest.mjs tests/Unit/ServerParityPayloadCleanupContractTest.mjs tests/Unit/ServerParityRepairPassContractTest.mjs tests/Unit/ServerParityNamespaceDeleteContractTest.mjs > parity-evidence/comparator-tests.txt
           docker run --rm --user=1000:1000 --cpus=1 --memory=256m --memory-swap=256m \
             --network=none --entrypoint node -v "$PWD:/source:ro" -w /source \
             "$image" --test tests/Unit/ServerParityCodecDeadlineLoadTest.mjs > parity-evidence/codec-load-tests.txt

docs/rust-server-port-checkpoints/2026-10-22-payload-cleanup.md

@@ -0,0 +1,50 @@
+# October 22, external payload cleanup
+
+- **Source.** Server 2.11.1 `SystemController::externalPayloadCleanupStatus`
+  and `externalPayloadCleanupPass`, `RuntimeExternalPayloadCleanup::runPass`,
+  `RuntimeExternalPayloadCleanupMetrics::snapshot`,
+  `RuntimePayloadCompletionUploads::cleanup`, `RuntimePayloadCompletionLease::
+  mayResume` and `ExternalPayloadBackupHold::deleting`.
+- **Served.** Native answers admin-only
+  `GET /api/system/external-payload-cleanup` and
+  `POST /api/system/external-payload-cleanup/pass` in the default or a named
+  namespace.
+  - The status is the cleanup section native's operator metrics already
+    report: the expired backlog, batch bounds, totals and the last pass.
+  - The pass takes `limit` (`nullable|integer|min:1|max:1000`, default 100).
+    It first visits the namespace's expired completion-upload allowances: one
+    whose activity can still complete (its task is leased to the same owner
+    and its attempt can continue) gets another abandoned-upload period, and
+    any other is dropped. It then reclaims the namespace's expired, unclaimed
+    references, oldest first, each under its object lock. A reference that
+    shares its object with another loses only its row; a sole reference also
+    deletes its object. The report lists processed, deleted, preserved,
+    skipped and blocked references, the remaining backlog and scan pressure,
+    and answers 207 when anything was blocked. Each pass is added to the
+    namespace's totals, with whole-second times as Eloquent stores them.
+- **Decisions.**
+  - Native stores objects only with the local driver. A reference without an
+    enabled local policy, or whose object lies outside the current local root,
+    is blocked rather than deleted. PHP would delete through its configured
+    remote driver or old root.
+  - Native never takes an online-backup hold. A hold row left active blocks
+    deletion as in PHP; a missing row means no hold, where PHP would report a
+    storage failure because its migration seeds the row.
+  - Native's completion allowances hold activity contexts only. An allowance
+    whose context native cannot read cannot resume and is dropped once
+    expired; PHP would fail the pass on a context it cannot parse.
+- **Fixture.** `payload-cleanup` checks a fresh namespace's status, a pass,
+  the recorded totals, limit validation (0, 1001, text and a fraction answer
+  422; null and numeric text are accepted) and the final count of five
+  passes. Deletion needs expired rows and stored objects, which an HTTP probe
+  cannot make within the abandoned-upload period, so native tests cover it on
+  SQLite, PostgreSQL and MySQL: shared, live and retained references, the
+  limit and its backlog, an active backup hold, a disabled policy, and an
+  allowance renewed while its activity runs and dropped after it completes.
+- **Evidence.** The new tests pass on SQLite, PostgreSQL and MySQL, and the
+  native suite with `--all-targets` passes on SQLite (652 tests). A local
+  copy of the SQLite parity step passed all 54 fixtures on embedded PHP and
+  native, including `payload-cleanup`. The published PHP fixture answers
+  were checked against a fresh 2.11.1 server. Its record differed only on
+  `namespace-delete`, a retryable 503 `backend_lock_pressure` that its probe
+  did not retry; that probe and the storage diagnostic now retry it.

docs/rust-server-port-checkpoints/2026-10-22-prometheus-metrics.md

@@ -0,0 +1,58 @@
+# October 22, Prometheus metrics summary
+
+- **Source.** Server 2.11.1 `SystemController::prometheusMetrics` and
+  `PrometheusMetricsSummary::snapshot`, with `BoundedMetricPolicy` and the
+  declared dimensions in `config/dw-bounded-growth.php`.
+- **Served.** Native answers admin-only `GET /api/system/prometheus-metrics`
+  in the default or a named namespace:
+  - workflow series by task queue and workflow type: started, completed,
+    failed, cancelled and terminated totals, running runs and a latency
+    histogram from run summaries;
+  - activity series by task queue, workflow type and activity type, with the
+    same counters and a latency histogram from activity executions;
+  - task-queue runtime state: ready, due, delayed, leased and expired tasks,
+    tasks added and dispatched in the last minute, and fresh worker pollers,
+    slot capacity and available slots.
+
+  Each family keeps the first 100 series in PHP's order. Native reads at
+  most 101 candidate label sets, then aggregates only the series up to the
+  last candidate. The disclosure reports observed, reported and suppressed
+  series and the suppressed starts, with `lower_bound` precision once the
+  limit is reached. Native refuses the series-limit settings, so the limit is
+  always PHP's default of 100.
+- **Times.** Task and worker comparisons bind whole seconds, as Laravel does
+  and as native's operator metrics already do. A task made available within
+  the current second still counts as delayed. Workers are fresh within
+  `DW_WORKER_STALE_AFTER_SECONDS`, by default `max(3 × heartbeat, 30)`.
+- **Decision: cumulative latency buckets.** Published PHP binds each bucket
+  bound as text. SQLite orders every number before every text value, so PHP
+  on SQLite counts every duration in every bucket
+  ([server#683](https://github.com/durable-workflow/server/issues/683)).
+  Native compares durations with numeric bounds and reports the cumulative
+  histogram on every backend. The fixture accepts PHP's SQLite shape as a
+  recorded reference defect; native's tests pin the cumulative buckets with
+  fixed durations on SQLite, PostgreSQL and MySQL.
+- **Correction: undispatched activity tasks.** PHP's worker completion
+  publishes only the timer tasks it creates. An activity task scheduled by a
+  workflow task therefore keeps null `last_dispatched_at` and
+  `last_dispatch_attempt_at` until a poller claims it, and its queue's
+  dispatched-in-the-last-minute counter stays 0. Native stamped it
+  dispatched. It now leaves those columns null for that path, as PHP does.
+  Other task paths keep their stamps.
+- **Fixture.** `prometheus-metrics` reads a fresh namespace, then:
+  - registers a worker with four workflow and three activity slots on the
+    first queue;
+  - completes a run there after one activity;
+  - terminates a second run there;
+  - starts a run on a second queue that has no worker.
+
+  Counters, task-queue state, worker capacity and the disclosure are exact.
+  Each counted run and activity lasts at least 1.1 seconds, so latency sums
+  and the empty low buckets are checked against that bound.
+- **Evidence.** Native tests pass on SQLite, PostgreSQL and MySQL, and the
+  native suite with `--all-targets` passes on SQLite (649 tests).
+  A local copy of the SQLite parity step passed all 53 fixtures on embedded
+  PHP and native, including `prometheus-metrics`. Published PHP passed
+  `prometheus-metrics` too; its only difference was `storage-test`, an
+  intermittent 503 from PHP's external payload registration on that loaded
+  host, which passed in the previous local run.

docs/rust-server-port-checkpoints/2026-10-23-activity-heartbeats.md

@@ -0,0 +1,26 @@
+# October 23, activity heartbeats on a closed run
+
+- **The gap.** When a run was terminated or cancelled while its activity
+  ran, published Server 2.11.1 answers the activity's next heartbeat with
+  200:
+  - `cancel_requested: true` and `can_continue: false`;
+  - the closing reason, such as `run_terminated`, with
+    `run_closed_reason` and `run_closed_at`;
+  - `heartbeat_recorded: false` and a null `lease_expires_at`.
+
+  The worker then stops the activity as cancelled. Native answered 409
+  with its activity-outcome body, which a worker can read as a transport
+  error rather than a cancellation.
+- **What native does now.** For a closed run, native answers PHP's
+  structured 200 from the same observation its status route reads.
+  Nothing is recorded and no lease is renewed. An ordinary heartbeat also
+  carries PHP's fields:
+  - `reason`, `run_closed_reason` and `run_closed_at`, all null;
+  - `worker_session`, null;
+  - `protocol_version` and `server_capabilities`.
+- **How it was checked.** A probe ran on native and the published image.
+  It scheduled an activity with all four timeouts, claimed it, sent a
+  heartbeat, terminated the run, and sent another heartbeat. The claim's
+  `deadlines` map and both heartbeat bodies matched.
+- **Not checked yet.** A heartbeat while a cooperative cancellation is
+  pending on a run that is still open.

docs/rust-server-port-checkpoints/2026-10-23-activity-lease-expiry.md

@@ -0,0 +1,63 @@
+# October 23, activity lease expiry
+
+- **Source.** Server 2.11.1 `ActivityTaskPoller::claimReadyTask` and
+  `ActivityTaskClaimer`, `TaskRepair::recoverLeasedTask`,
+  `ActivityTimeoutEnforcer::enforceLeaseExpiry` (through
+  `enforceLockedTimeout`, `leaseLossMetadata`, `closeTimedOutAttempt`,
+  `settleTimeout`, `scheduleRetry` and `recordTerminalTimeout`) and
+  `ActivityRetryPolicy`.
+- **Published behavior.** PHP claims only ready activity tasks, so a worker
+  never receives a task again after its lease expires. PHP's maintenance runs
+  the repair pass every second, and the pass settles each expired activity
+  lease:
+  - The original attempt closes as expired, and its task is cancelled with
+    `last_error` `lease_expired`.
+  - When the activity's retry policy has an attempt left, PHP records
+    `ActivityRetryScheduled`. It carries `retry_reason` and `failure_reason`
+    `lease_expired`, the lease's owner and expiry, and no `timeout_kind`.
+    The retry task waits out the policy's backoff.
+  - Otherwise PHP records a terminal `ActivityFailed` with
+    `Workflow\V2\Exceptions\ActivityLeaseExpiredException`, failure category
+    `internal` and propagation `activity`. Its `non_retryable` is true when
+    the policy lists that exception. The workflow then runs again to see the
+    failure.
+  - An activity whose own deadline also expired is left to the timeout sweep.
+  - Without a retry policy an activity has one attempt.
+- **Evidence for PHP.** A fresh 2.11.1 server on SQLite, with its
+  one-second repair loop, claimed two activities and had their leases
+  expired in the database. The activity without a policy recorded
+  `ActivityFailed` as above. The one with two attempts recorded
+  `ActivityRetryScheduled` with `retry_reason` `lease_expired`, then attempt
+  two started on a new task.
+- **Native before.** Native's claim query took a leased activity task whose
+  lease had expired and ran it again as the next attempt. That skipped the
+  retry policy, its backoff and the history events above. An activity
+  without retries ran twice.
+- **Served.** Native's claim query and its cross-kind poll take only ready
+  activity tasks. Native's scheduler settles expired activity leases on each
+  tick, after its deadline sweep, exactly as PHP's enforcer does. The repair
+  pass now settles an expired running-activity lease the same way; it no
+  longer reports it as unported. The same check against native recorded the
+  same events, classes, reasons, lease fields and attempt numbers as PHP.
+  - Workflow task leases are unchanged: PHP's workflow poller also takes a
+    workflow task whose lease expired.
+- **Decisions.**
+  - PHP's lease events also embed the closed task's snapshot. Native's
+    activity events do not carry task snapshots yet; this is the same known
+    payload difference as its timeout events.
+  - A run that already closed is skipped as `run_already_terminal`. Native
+    never delivers an activity task of a closed run, so it does not continue
+    an abandoned activity there.
+- **Restart probe contract.** `restart.php` kills a native process while it
+  holds an activity lease, waits out the real five-minute lease and finishes
+  with a new worker. It expected the recovered worker to run attempt two of
+  an activity without a retry policy, with no event in between. That
+  expectation came from native's re-claim, not from PHP: published PHP fails
+  that activity instead. The probe now declares two attempts for the
+  activity in `current-activity-restart.json` (`activity_retry_policy`) and
+  expects `ActivityRetryScheduled` with `retry_reason` `lease_expired`
+  between the two `ActivityStarted` events.
+- **Tests.** Native tests on SQLite, PostgreSQL and MySQL cover a retry,
+  a terminal failure, a non-retryable lease exception, a poll racing the
+  repair loop, the repair pass and an external result refused after the
+  lease expired.