Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 35 additions & 19 deletions apps/job_runner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ result archives. Nomad is not used.

| Component | Location | Supervision |
| --- | --- | --- |
| Dashboard and API | `/mnt/server_ssd/lhr-jobs/app` on telemetry | PM2 app `lhr-jobs`, Node >=18 |
| Dashboard and API | `/mnt/server_ssd/lhr-jobs/app` on telemetry | PM2 app `lhr-jobs` under `electric`, Node >=18 |
| Dashboard configuration/key | `~/.config/lhr-jobs` on telemetry | User-only files; never committed |
| Runner executable | `/opt/lhr-runner` on HIL | Root-owned code, systemd runs as `lhr-runner` |
| Queue, logs and results | `/var/lib/lhr-runner` on HIL | SQLite WAL and per-run directories |
Expand All @@ -22,12 +22,17 @@ bridge. The runner exposes no network listener or Docker TCP port. Docker socket
access stays on HIL. Browser credentials and worker SSH credentials are
separate; neither is sent to jobs.

The dashboard listens on `127.0.0.1:8091`. A separate private Tailscale Serve
HTTPS listener on port 8444 forwards to it. This port cannot be enabled for
Funnel. The existing public Funnel on 443, telemetry viewer, Grafana, ingestion
and database services keep their own routes. Check `tailscale serve status
--json` after changes: the new 8444 listener must not appear as enabled under
`AllowFunnel`.
The initial deployment listens on `127.0.0.1:8091` only. Use an SSH tunnel
from the operator's machine:

```bash
ssh -N -L 127.0.0.1:8091:127.0.0.1:8091 electric@telemetry
```

Open `http://127.0.0.1:8091`; `publicOrigin` must match that local URL. No new
Tailscale Serve listener or domain route is installed. Telemetry's existing
public Funnel on 443, viewer, Grafana, ingestion and database services keep
their routes. A separate HTTPS listener can be configured later when requested.

## Workloads and limits

Expand All @@ -51,11 +56,15 @@ have no network, a read-only root filesystem, dropped capabilities, no Docker
socket or writable host mounts. Swap is disabled by setting the memory-plus-swap
limit equal to the memory limit. Missing local images fail with `--pull=never`;
a job never pulls a replacement from Docker Hub. GPU access is granted only to
the GPU template. Jobs stop after 15 minutes.
the GPU template. Jobs stop after 15 minutes. FMU scratch is explicitly
executable because FMPy extracts native shared libraries there before loading
them. It remains a 512 MiB in-memory mount with `nosuid,nodev`; CPU/GPU
templates retain Docker's default non-executable scratch mount.

The worker process itself has a 256 MiB memory cap and 25% of one CPU; Docker
job limits are independent. PM2 restarts the dashboard above 256 MiB RSS and
Node's JS heap is limited to 128 MiB. A PM2 restart threshold is not a hard
Node's old generation is limited to 128 MiB and its semi-space to 8 MiB.
A PM2 restart threshold is not a hard
operating-system memory or CPU quota. Measure actual usage with `pm2 list` and
`systemctl status lhr-runner`.

Expand All @@ -68,7 +77,9 @@ bound archive decoding memory and serialize cache pruning. New jobs are refused
below 2 GiB free on HIL. Do not put outputs under HIL's unconfigured `/data`
mount.

SQLite is authoritative for queue/history. IDs make retrying a submission
SQLite is authoritative for queue/history. Each transaction closes its owned
connection after commit or rollback, so the long-running worker does not depend
on garbage collection to release database file handles. IDs make retrying a submission
idempotent. Worker restarts reconcile deterministic container names instead of
starting duplicates. Dashboard restarts or SSH disconnects do not stop
containers. Cancellation is reported complete only after Docker confirms the
Expand Down Expand Up @@ -97,8 +108,10 @@ are included in stdout. The workloads have no writable host output mount, so
they cannot fill HIL's disk through `/results`. Detailed older output may rotate
out; these are smoke/timing templates, not general artifact-producing studies.

The dashboard runs as a dedicated `lhr-jobs` user without Docker/sudo access.
Its SSH key logs in as `lhr-job-ssh`, which may invoke only the bridge as
The initial dashboard runs as the existing `electric` account alongside
`viewer_tool`, as recorded in [ADR-020](../../docs/architecture/020-shared-telemetry-dashboard-account.md).
It inherits that account's existing Docker/admin access. Its separate worker
key logs in as `lhr-job-ssh`, which may invoke only the bridge as
`lhr-runner`. The worker's Docker group is administrator-equivalent on HIL; this
is a trusted service account, not a Docker privilege boundary. At most four
bridge RPCs and two dashboard SSH requests run concurrently. Eight additional
Expand All @@ -109,9 +122,10 @@ logs each worker cycle; both are sampled by the supervised worker; read RPCs do
not invoke Docker or NVIDIA. The dashboard coalesces log reads for 2.5 seconds.
Sign-in checks allow five failed credential checks per source per minute; cached
valid logins bypass that limit. Empty credentials always receive a 401 browser
challenge without using an attempt. Sources are the socket peer, or the source
IP supplied by Tailscale Serve over the loopback proxy. Different tailnet
clients have separate limits. If HIL cannot be reached, the dashboard labels
challenge without using an attempt. SSH-tunneled browser sessions share
telemetry's loopback rate limit. When Tailscale Serve is configured as the
trusted loopback proxy, the forwarded source IP gives each client its own
limit. If HIL cannot be reached, the dashboard labels
cached history as disconnected and disables new submissions.

## Development and checks
Expand All @@ -129,7 +143,7 @@ inject fixed times. They cover bounds/injection, authentication, CSRF,
idempotency, pause/cancel, restart reconciliation, resource slots and archive
symlinks, launch timeouts, per-job failures, transport decoding/limits, login
throttling, archive retention, browser focus, stale-state controls, workload
switching and fixed polling. CPU and FMU tests fake clocks and subprocesses
switching, fixed polling and native PM2 entrypoint startup. CPU and FMU tests fake clocks and subprocesses
without real benchmark files. Linux runs the runner tests through Bazel; CI also
runs Node tests. Real host/container tests are a separate deployment check,
never unit tests.
Expand All @@ -146,11 +160,13 @@ host verification.
Deployment source and service templates are described in
[deploy](deploy/README.md). Keep keys, password hashes, databases, model
binaries and outputs outside the checkout. Pin the source FMU and base runtime
image in the deployment record before enabling a new workload.
image in the deployment record before enabling a new workload. The PM2
template executes Node directly in fork mode: PM2's JavaScript wrapper changes
`argv[1]` and otherwise bypasses this app's ESM entrypoint guard.

To stop this service without touching telemetry: `pm2 stop lhr-jobs` on
telemetry, pause the worker queue, and `sudo systemctl stop lhr-runner` on HIL.
Stopping the worker alone leaves already-running job containers running; cancel
those jobs first if they should stop. Remove only the 8444 Serve listener when
withdrawing browser access. Do not reset Tailscale Serve or stop unrelated
those jobs first if they should stop. Close the operator's SSH tunnel when
withdrawing local browser access. Do not reset Tailscale Serve or stop unrelated
Compose projects.
52 changes: 52 additions & 0 deletions apps/job_runner/deploy.test.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
import {test} from 'node:test';
import assert from 'node:assert/strict';
import {createRequire} from 'node:module';
import {mkdtemp, writeFile, rm} from 'node:fs/promises';
import {tmpdir} from 'node:os';
import path from 'node:path';
import {spawnSync} from 'node:child_process';
import {fileURLToPath} from 'node:url';

const require = createRequire(import.meta.url);
const {apps: [application]} = require('./deploy/ecosystem.config.cjs');

for (const [label, override, expected] of [
['unset', undefined, '/home/electric/.config/lhr-jobs/dashboard.json'],
['empty', '', '/home/electric/.config/lhr-jobs/dashboard.json'],
['custom', '/tmp/custom-dashboard.json', '/tmp/custom-dashboard.json'],
]) {
test(`PM2 selects the dashboard config when LHR_JOBS_CONFIG is ${label}`, () => {
const configPath = fileURLToPath(new URL('./deploy/ecosystem.config.cjs', import.meta.url));
const child = spawnSync(process.execPath, [
'-e', 'console.log(require(process.argv[1]).apps[0].env.LHR_JOBS_CONFIG)', configPath,
], {
env: override === undefined ? {} : {LHR_JOBS_CONFIG: override},
encoding: 'utf8',
timeout: 10000,
});
assert.equal(child.status, 0, child.stderr || child.error?.message);
assert.equal(child.stdout.trim(), expected);
});
}

test('PM2 launches a native Node process whose ESM entrypoint guard runs', async t => {
const directory = await mkdtemp(path.join(tmpdir(), 'lhr-native-entrypoint-'));
t.after(() => rm(directory, {recursive: true, force: true}));
await writeFile(path.join(directory, 'server.mjs'), `
import path from 'node:path';
import {fileURLToPath} from 'node:url';
import v8 from 'node:v8';
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
console.log(JSON.stringify({started: true, heapLimit: v8.getHeapStatistics().heap_size_limit}));
}
`);
// PM2's JavaScript wrapper changes argv[1]; native execution preserves the entrypoint.
assert.equal(application.interpreter, 'none');
assert.equal(application.exec_mode, 'fork');
assert.equal(application.instances, 1);
const child = spawnSync(application.script, application.args, {cwd: directory, encoding: 'utf8', timeout: 10000});
assert.equal(child.status, 0, child.stderr || child.error?.message);
const result = JSON.parse(child.stdout);
assert.equal(result.started, true);
assert.ok(result.heapLimit < 256 * 1024 * 1024);
});
86 changes: 86 additions & 0 deletions apps/job_runner/deploy/2026-11-14.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Initial live deployment — 2026-11-14

The dashboard runs under telemetry's existing `electric` PM2 daemon as
`lhr-jobs`. The HIL worker runs under systemd as `lhr-runner`, with a separate
`lhr-job-ssh` account and forced-command key. Base source is merged commit
`cde7e404`, plus the PM2 native-launch, FMU scratch and SQLite ownership fixes in this
change.

## Access and persistence

The HTTP listener is `127.0.0.1:8091` on telemetry. An operator SSH tunnel
provides local browser access. No domain, new Tailscale Serve listener or
Funnel setting was added. The previous public route is unchanged.

The configuration and dedicated worker key live under
`/home/electric/.config/lhr-jobs`, outside the checkout. Bootstrap credentials
are stored there in a mode-0600 operator file; no credentials appear in this
record. The HIL host key was obtained over the existing verified SSH session
and installed in the dashboard's dedicated known_hosts file.

The worker is enabled at boot. PM2's saved process list includes both
`viewer_tool` and `lhr-jobs`. Telemetry still has its existing PM2 startup
behavior: no new system-level PM2 startup unit was installed. Saved process
configuration alone does not guarantee dashboard startup after a host reboot.

## Acceptance

| Check | Observed result |
| --- | --- |
| Authenticated UI and health | HTTP 200 through telemetry loopback and the local SSH tunnel |
| Restricted SSH bridge | Status RPC succeeds with the dedicated key and strict host verification |
| CPU smoke | Five-second job succeeds; archive contains final metadata, JSON summary and both logs |
| Queued cancellation | Job becomes cancelled without starting a container |
| Running cancellation | Container stops and job becomes cancelled |
| Pause/resume | Queued starts pause; persisted pause survives both service restarts; queue left resumed |
| Restart recovery | A running 90-second CPU job keeps its container ID through worker restart and succeeds after dashboard restart |
| FMU timing | One profile completes all 12 simulated seconds in 80.563 wall seconds; weighted step mean 132.508 ms |
| GPU check | Existing NVIDIA rendering-check image exits successfully; logs downloadable |
| Existing telemetry | All 11 original container IDs/states, viewer process and public route unchanged |
| Database lifetime | 4,000 database operations leave descriptor count at 4; the running worker stays at 4–6 descriptors over 30 seconds |
| Permissions | Runner has Docker membership; SSH account does not; code/bridge/sudoers root-owned; state mode 0700 |

The first FMU attempt failed because Docker mounted scratch with `noexec`.
After applying the template-specific executable tmpfs fix, the full FMU profile
and an independent native-library load probe pass. The failed preflight remains
in history rather than being erased. Solver diagnostics remain in the logs;
this validates standalone job execution, not full autonomy/Gazebo accuracy.

The first PM2 launch imported the server without starting its HTTP listener.
The native Node/fork configuration fixes this; a PM2 "online" status alone is
not an acceptance check. Both failures have hermetic regression tests.

The longer idle check also found a SQLite connection leak: transaction context
exit committed/rolled back but did not close connections. The worker eventually
exhausted file handles. Each transaction now explicitly closes its connection,
including on rollback. A deterministic ownership test and the live descriptor
soak above verify the fix. A further CPU job and archive download pass after
the soak.

## Artifacts and resources

Runtime image: `lhr/job-runtime:2026-11-14-stdout-v2`, image ID beginning
`44e3220c7323`; existing GPU image ID begins `ec734ec4744f`. The native FMU
SHA-256 remains
`b6d3d4ddc8c451d421a95704951f642ea0a8eaf17414f85633d57e4ff5aa71c9`.

Full private records stay on the hosts:

- HIL: `/opt/lhr-runner/deployment.json` and
`/var/lib/lhr-jobs-deployment/cde7e404/`.
- Telemetry: `/mnt/server_ssd/lhr-jobs/app/deployment.json` and
`/mnt/server_ssd/lhr-jobs/data/deployment-acceptance.json`.

A 30.03-second idle sample measured the HIL worker's service group at
14.65 MiB and 2.73% of one CPU core, including its monitoring child processes.
The configured limits remain 256 MiB and 25% of one core. A separate
30-second idle dashboard sample measured 58.13 MiB RSS and 0.0% of one core
with no active browser polling. Container workload limits are separate. These
short samples are not sustained-load benchmarks.

## Rollback

On telemetry, stop only `lhr-jobs` in PM2 and close the operator SSH tunnel.
On HIL, pause/cancel jobs first when they should stop, then stop/disable only
`lhr-runner`. Worker shutdown alone leaves Docker jobs running. Do not stop
`viewer_tool`, reset Tailscale Serve or modify the existing Compose projects.
Loading