Node `endGlobal` never runs for chat/webhook-sourced pipelines, so nodes cannot release external resources

[2001anshkaushik]

Summary

A Python node's endGlobal is never called when a pipeline whose source is chat or webhook is
stopped, on either of the two normal stop paths: an explicit terminate(), and idle-TTL expiry. The
task process is force-killed before node teardown runs.

Nodes that hold an external, billable resource use endGlobal to release it. For those nodes the
resource is simply leaked when the pipeline stops, and stays alive until the vendor's own timeout
ends it.

Measured on engine 3.3.0 (built from develop, macOS arm64).

Impact

Three nodes in this repo release external state in endGlobal:

Node	Released in endGlobal	What leaks when it doesn't run
tool_daytona	sandbox.delete()	A Daytona sandbox, until its auto-stop interval
db_hotdata	database teardown	A billed database
tool_tenki	session.close() plus a tagged sweep	A Tenki VM, until its idle pause and max_duration

Every one of these is an agent tool, and agent tools are used from chat-sourced pipelines. So the
affected path is the normal one, not an edge case.

Severity is "costs money quietly" rather than "breaks a run": each vendor has a server-side timeout,
so the leak is bounded. It is invisible to the operator, though, and it makes the documented
endGlobal contract untrue for the most common pipeline shape.

Reproduction (no vendor account needed)

A probe node that records its own lifecycle to disk. fsync is deliberate, so the record survives a
force-kill of the process.

local_nodes/probe_teardown/IGlobal.py:

from __future__ import annotations

import os
import time

from rocketlib import IGlobalBase

MARKER = os.environ.get('PROBE_TEARDOWN_MARKER', '')


def _mark(event: str) -> None:
    if not MARKER:
        return
    with open(MARKER, 'a') as handle:
        handle.write(f'{time.time():.3f} {event} pid={os.getpid()}\n')
        handle.flush()
        os.fsync(handle.fileno())


class IGlobal(IGlobalBase):
    def beginGlobal(self) -> None:
        _mark('beginGlobal')

    def endGlobal(self) -> None:
        _mark('endGlobal')

with services.json declaring "path": "local_nodes.probe_teardown", "classType": ["tool"], and a
matching IInstance subclassing IInstanceBase.

Steps:

1. Start the engine with the probe on the node path:

   PROBE_TEARDOWN_MARKER=/tmp/probe.log ./engine ai/eaas.py \
       --host=127.0.0.1 --port=5578 --node_path=/path/to/dir-containing-local_nodes

2. Start a pipeline chat -> agent_rocketride -> (llm, memory_internal, probe_teardown) -> response_answers.
   No question needs to be asked; the LLM key can be a dummy.
3. Case A: call terminate(token).
   Case B: start with ttl=60 and let the idle TTL stop it.
4. Read /tmp/probe.log.

Result

Stop path	beginGlobal	endGlobal
Explicit terminate()	1	0
Idle TTL expiry	1	0

1789544634.994 beginGlobal pid=30544   <- case A, terminate
1789544667.314 beginGlobal pid=30629   <- case B, idle TTL

beginGlobal appearing is the control: the marker mechanism works and the node loaded, so the
missing endGlobal is a real absence rather than a broken probe. Both cases were run against the
same engine process, one after the other.

terminate() returned after 5.0 s, matching CONST_CANCEL_WAIT_TIMEOUT_SECONDS.

The task process reports the signal itself, from its own console output:

Application received system signal: SIGTERM: Termination request (15)
Cancel request, fail safe in 10s

Observed with a real resource

The same behaviour seen end to end with tool_tenki: after terminate(), its Tenki VM stayed
RUNNING for the full 90 s we polled, and had to be closed out of band. Node teardown, which would
have closed it immediately, never ran.

Where it appears to come from

Reading the code, the chain looks like this. The first two points are confirmed by the measurements
above; the rest is code reading and should be checked by someone who knows the C++ side.

1. stop_task sends SIGTERM, waits CONST_CANCEL_WAIT_TIMEOUT_SECONDS (5), then engine.kill() —
   packages/ai/src/ai/modules/task/task_engine.py (~2348-2395), packages/ai/src/ai/constants.py:49.
2. In the task process, SIGTERM sets a cancel flag with a 10 s fail-safe exit —
   packages/server/engine-lib/engLib/application/unx/signal.cpp (~81-91), async/api.cpp (~35-60).
3. A Python source only observes the cancel when it calls the scan callback —
   packages/server/engine-lib/engLib/store/python/python-endpoint.source.cpp (~47-50).
4. The chat/webhook source never does. scanObjects runs the web server and blocks on an Event that
   nothing sets — nodes/src/nodes/webhook/IEndpoint.py (~186-217). chat is the same module
   (services.chat.json has "path": "nodes.webhook").
5. So the scan never returns, endTask never runs, and the 5 s kill arrives first. endGlobal is
   only reachable through endTask -> endEndpoint -> endFilterGlobal —
   engLib/store/python/python-global.cpp (~154-174), engLib/store/core/endpoint/endpoint.cpp (~104-146).
6. ITask::execute also only reaches endTask when exec() succeeded — engLib/task/core/task.cpp
   (~95-102) — so even a source that did return on cancel might skip teardown, because the run ends
   as cancelled.

Point 6 is the part we are least sure about, and it decides whether fixing the source is enough.

What we are not asking for

We are not proposing a fix in a node. This is engine signal handling and we would rather report it
than work around it. A node-side watchdog would live in the same process that gets killed, so it
could not help anyway.

Two directions a maintainer might weigh:

• Have the chat/webhook source observe the cancel so its scan returns, and make sure a cancelled run
  still reaches endTask.
• Or give nodes a teardown hook that runs on the cancel path, before the fail-safe exit.

If the answer is "by design, use vendor timeouts", that is a fine outcome — but the node docs should
say so, because today they promise cleanup that does not happen.

Workaround available to node authors today

Only the vendor's own server-side timeout. For tool_tenki that is Tenki's idle pause (stops
compute) and max_duration (ends the session). Both are set at create time, so they survive the
task being killed. Node documentation should say this rather than promising teardown; we corrected
tool_tenki's README accordingly.

Environment

• engine 3.3.0, built from source on develop (macOS 26.6, arm64)
• Reproduced twice, on two stop paths, in one engine session
• Probe node, engine log and the end-to-end transcript are available on request

---

Adjacent, not part of this report

While investigating we also noticed that a tool call carries no caller identity
(IInvokeTool.Invoke has lane, op, tool_name, input, output), and instances are pooled and
reused across callers. That is what stops a tool node from isolating one caller's resources from
another's in a team-deployed pipeline. It is a separate design question from this bug; mentioning it
only because both surfaced from the same lifecycle work, and a fix here might touch the same code.

[github-actions]

🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.