Merge pull request #891 from durable-workflow/fix/878-embedded-activity-maintenance

Maintain embedded activity leases without moving callbacks

Author: rmcdaniel

README.md

@@ -130,6 +130,7 @@ Server or Cloud does not reinterpret existing embedded history.
 - [Embedded installation](https://durable-workflow.com/docs/2.0/installation/)
 - [Embedded feature guides](https://durable-workflow.com/docs/2.0/category/embedded/)
 - [Embedded live execution](docs/workflow/embedded-live-execution.md)
+- [Embedded activity leases](docs/workflow/embedded-activity-leases.md)
 - [Configuration reference](https://durable-workflow.com/docs/2.0/configuration/options/)
 - [Deployment modes](https://durable-workflow.com/docs/2.0/polyglot/deployment-modes/)
 - [Monitoring with Waterline](https://durable-workflow.com/docs/2.0/monitoring/)

docs/workflow/embedded-activity-leases.md

@@ -0,0 +1,50 @@
+# Embedded activity leases
+
+A lease records which attempt owns an activity. It is separate from the activity's
+start-to-close, schedule-to-close and heartbeat timeouts, and from workflow run
+and execution timeouts.
+
+For committed activities on persistent storage, the runtime starts an independent
+PHP metadata process before invoking the activity. The callback stays in its
+original PHP process, with its Laravel services, captured state and connections.
+The helper renews only that original attempt and stops when its owner exits.
+It is joined before the callback outcome is recorded.
+
+The helper uses the original database connection settings, including its TLS and
+authentication configuration. Automatic maintenance needs `proc_open` and an
+available PHP CLI executable. Laravel's existing Symfony executable finder locates the CLI binary
+when the activity is invoked from another PHP SAPI. No activity arguments,
+results or exception payloads are sent to the helper.
+
+SQLite maintenance uses the database's current journal mode. Opening the helper
+does not replay a journal-mode setting that the running application has changed.
+
+If process creation or a PHP CLI binary is unavailable, the callback keeps its
+original captured lease and deadline. Short activities retain their previous
+behavior, but automatic renewal is unavailable and grants no additional time.
+
+Runtime renewal records no heartbeat event and supplies no application progress.
+Activities with a heartbeat timeout must still call `heartbeat()` with genuine
+progress. Renewal cannot extend a recorded business deadline, revive an expired
+grant, transfer the attempt to another worker, or authorize a late result.
+Brief database contention retries only inside the last confirmed grant. Loss of
+authority fences the outcome; it cannot interrupt arbitrary blocking PHP code or
+undo an external side effect.
+
+`DW_V2_ACTIVITY_TASK_LEASE_SECONDS` configures the ownership lease, with a default
+of 300 seconds and valid values from 1 through 86400. Increasing it also increases
+how long recovery may wait after an unannounced worker death. It is not an
+activity execution timeout. Keep the usual worker, repair and timeout processing
+running so expired attempts receive a durable recovery outcome.
+
+In-memory databases, uncommitted outer transactions, mocked activities and a
+frozen application test clock have no independently observable live grant. They
+retain the captured lease without an independent helper. Use persistent storage
+and a real clock when qualifying long-running queue workers. Custom database
+connectors must be available to the fresh metadata process; application-specific
+connector bootstraps are not qualified by the built-in driver tests.
+
+Legacy activity jobs that start after their workflow has completed or failed
+retain the original captured lease and outcome handling. They receive no new
+renewal authority. An activity retained by the runtime's explicit abandon policy
+can renew only within that policy's original recorded deadlines.

src/V2/Jobs/RunActivityTask.php

@@ -16,6 +16,7 @@
 use Workflow\V2\Support\ActivityRetryPolicy;
 use Workflow\V2\Support\ActivityTaskClaim;
 use Workflow\V2\Support\ActivityTaskClaimer;
+use Workflow\V2\Support\EmbeddedActivityLease;
 use Workflow\V2\Support\EntryMethod;
 use Workflow\V2\Support\ExecutionTracing;
 use Workflow\V2\Support\RuntimeObjectFactory;
@@ -100,6 +101,16 @@ public function handle(): void
     }
 
     private function executeClaim(ActivityTaskClaim $claim): void
+    {
+        $maintenance = EmbeddedActivityLease::start($claim);
+        try {
+            $this->executeMaintainedClaim($claim, $maintenance);
+        } finally {
+            $maintenance?->stop();
+        }
+    }
+
+    private function executeMaintainedClaim(ActivityTaskClaim $claim, ?EmbeddedActivityLease $maintenance): void
     {
         $attemptId = $claim->attemptId();
         $attemptCount = $claim->attemptNumber();
@@ -150,6 +161,8 @@ private function executeClaim(ActivityTaskClaim $claim): void
         }
         $backoffSeconds = ActivityRetryPolicy::backoffSeconds($execution, $activity, $attemptCount);
 
+        $maintenance?->confirm();
+
         $outcome = ActivityOutcomeRecorder::record(
             $this->taskId,
             $attemptId,

src/V2/Support/ActivityTaskClaim.php

@@ -38,6 +38,8 @@ final class ActivityTaskClaim
 
     private ?int $lastRenewalRequestStartedAt = null;
 
+    private ?int $maintainedAdmissionDeadline = null;
+
     public function __construct(
         public readonly WorkflowTask $task,
         public readonly WorkflowRun $run,
@@ -91,6 +93,11 @@ public function __construct(
      */
     public function outcomeAdmissionDeadline(): int
     {
+        if ($this->maintainedAdmissionDeadline !== null) {
+            return $this->businessAdmissionDeadline === null
+                ? $this->maintainedAdmissionDeadline
+                : min($this->maintainedAdmissionDeadline, $this->businessAdmissionDeadline);
+        }
         $deadline = $this->businessAdmissionDeadline === null
             ? $this->leaseAdmissionDeadline
             : min($this->leaseAdmissionDeadline, $this->businessAdmissionDeadline);
@@ -99,6 +106,45 @@ public function outcomeAdmissionDeadline(): int
             ? $deadline : min($deadline, $this->heartbeatAdmissionDeadline);
     }
 
+    /**
+     * @internal The independent maintainer validates every intermediate original grant.
+     */
+    public function maintenanceRequestDeadline(): int
+    {
+        $deadline = hrtime(true) + 5_000_000_000;
+
+        return $this->businessAdmissionDeadline === null ? $deadline : min($deadline, $this->businessAdmissionDeadline);
+    }
+
+    /**
+     * @return array<string, mixed>
+     */
+    public function identity(): array
+    {
+        return $this->identity;
+    }
+
+    /**
+     * @internal A fresh proof from this claim's private metadata process, never a remote Worker response.
+     * @param array<string, mixed> $reply
+     */
+    public function acknowledgeMaintenance(array $reply, int $started): bool
+    {
+        $received = hrtime(true);
+        if ($started < $this->claimedAt || $started > $received
+            || ($reply['can_continue'] ?? null) !== true
+            || ($reply['identity'] ?? null) !== $this->identity
+            || ! is_int($reply['monotonic'] ?? null) || $reply['monotonic'] < $started
+            || $reply['monotonic'] > $received || ! is_int($reply['until'] ?? null)
+            || $reply['until'] <= $received || $reply['until'] - $received > 86_400_000_000_000
+            || ($this->businessAdmissionDeadline !== null && $received >= $this->businessAdmissionDeadline)) {
+            return false;
+        }
+        $this->maintainedAdmissionDeadline = $reply['until'];
+
+        return true;
+    }
+
     /**
      * A successful transport response may extend only its original live claim.
      * Capture the request clock before renewal begins so its latency consumes budget.