fix(sdk): don't SIGKILL a database dump copy after 30 seconds

projects/start-sdk/CHANGELOG.md

@@ -75,6 +75,54 @@
   one, so the guard only duplicated that check while giving the filename a
   second place to go stale. Cosmetic — no build ever failed over it
 
+- **A database dump backup or restore is no longer killed after exactly thirty
+  seconds.** Steps in `Backups.withPgDump` / `Backups.withMysqlDump` run under
+  `SubContainer.exec`'s default 30 s cap unless they opt out, and 1.5.2 — which
+  began staging the dump in `/tmp` and copying it to the backup target, rather
+  than dumping onto the target directly — left the opt-out on `pg_dump` and
+  gave the new `cp` none. The copy's duration is set by the size of the dump and
+  the speed of the target. So a backup that had succeeded for months began
+  failing the first time that copy crossed thirty seconds, with nothing to point
+  at the cause:
+
+  ```
+  Failed: Unknown Error: Error: cp terminated with signal SIGKILL:
+  ```
+
+  Restore stages the dump off the target through the same kind of copy, under
+  the same cap — so a database whose dump took longer than thirty seconds to
+  copy could not be restored at all, which is discovered during recovery, when
+  the backup is all the user has. Every step of a dump or restore whose
+  duration follows the data now opts out: both copies, `pg_ctl start` and
+  `pg_ctl stop` (which wait on the cluster's crash recovery and its shutdown
+  checkpoint), the recursive `chown`s over the data directory, `initdb`,
+  `mysql_install_db` / `mysqld --initialize-insecure`, and the foreground
+  `mysqld` MariaDB runs for the length of the dump.
+
+  The two `pg_ctl` steps keep a bound, because `pg_ctl` has its own: `-w` is its
+  default and it gives up after `-t` seconds, which was 60 regardless of what
+  the SDK allowed. `PgDumpConfig.readyTimeout` — the knob 2.0.1 added for
+  clusters that need longer — now supplies that `-t`, so raising it reaches the
+  step that actually blocks instead of stopping at the readiness poll. Its
+  default is 60 000 ms, which is `pg_ctl`'s own default, so nothing changes
+  until you raise it. Fixes
+  [#3636](https://github.com/Start9Labs/start-technologies/issues/3636)
+
+- **A command killed by `SubContainer.exec`'s own timeout now says that it timed
+  out.** The error was built from the signal alone, so the SDK's timer read
+  exactly like an OOM kill, a cgroup kill, or an operator's `kill -9` — and
+  since `cp` writes nothing to stderr when it is killed, the whole user-facing
+  notification was `cp terminated with signal SIGKILL:`. The message now names
+  the timeout and the limit that elapsed:
+
+  ```
+  cp timed out after 30000ms and was killed with SIGKILL:
+  ```
+
+  `exec`'s result carries `timedOutAfterMs` — the limit that fired, or `null`
+  when the process was not killed by this timer — alongside `exitCode` and
+  `exitSignal`
+
 ## 2.0.9 — StartOS 0.4.0-beta.10 (2026-07-25)
 
 ### Fixed

projects/start-sdk/docs/src/main.md

@@ -477,7 +477,7 @@ if (result.exitCode !== 0) {
 
 // execFail() - throws on error (good for required commands)
 // Uses the default user from the Dockerfile (no need to specify { user: '...' })
-await appSub.execFail(['git', 'clone', 'https://github.com/user/repo.git'])
+await appSub.execFail(['myapp', 'check-config'])
 
 // Override user when needed (e.g., run as root)
 await appSub.exec(['update-ca-certificates'], { user: 'root' })
@@ -497,6 +497,23 @@ The `user` option is optional. If omitted, commands run as the default user defi
 - You need to inspect the exit code or output regardless of success/failure
 - You want custom error handling logic
 
+### Commands That Run Longer Than 30 Seconds
+
+`exec` and `execFail` both take a third argument, `timeoutMs`: how long the SDK waits before it gives up and fails the call. It defaults to **30 s**, so a command that legitimately takes longer — cloning a large repository, importing a database, copying a multi-gigabyte file — fails partway through unless you say otherwise. Pass `null` to wait as long as it takes:
+
+```typescript
+// Gives up after 30 s — fine for a command that either answers quickly or is stuck
+await appSub.execFail(['update-ca-certificates'], { user: 'root' })
+
+// No limit — takes as long as the database takes
+await appSub.execFail(['pg_restore', '-U', user, '-d', database, dumpFile], { user: 'postgres' }, null)
+```
+
+Opt out whenever the runtime is set by something you cannot bound: the size of the data, the speed of a disk or backup target, or another process you are waiting on. Keep the default for commands that should answer promptly, where the timeout is what stops a wedged container from hanging the service.
+
+> [!NOTE]
+> On timeout the SDK sends `SIGKILL` to the process it spawned and reports `timed out after <n>ms and was killed with SIGKILL`; `exec()`'s result carries `timedOutAfterMs`, set to the limit that elapsed. The command itself runs inside the subcontainer and is not signalled — it stops when the subcontainer is torn down, so treat a timeout as "the SDK stopped waiting", not "the work stopped".
+
 ## PostgreSQL Sidecar
 
 Many services require a PostgreSQL database. Run it as a sidecar daemon within the same service package.

projects/start-sdk/lib/backup/Backups.ts

@@ -12,6 +12,8 @@ import {
 
 const BACKUP_HOST_PATH = '/media/startos/backup'
 const BACKUP_CONTAINER_MOUNT = '/backup-target'
+// these steps can outlast exec's 30 s default
+const NO_TIMEOUT = null
 
 /** A password value, or a function that returns one. Functions are resolved lazily (only during restore). */
 export type LazyPassword = string | (() => string | Promise<string>) | null
@@ -41,7 +43,7 @@ export type PgDumpConfig<M extends T.SDKManifest> = {
   initdbArgs?: string[]
   /** Additional options passed to `pg_ctl start -o` (e.g. '-c shared_preload_libraries=vectorchord'). Appended after `-c listen_addresses=`. */
   pgOptions?: string
-  /** Milliseconds to wait for PostgreSQL to accept connections before failing (default 60000). Raise for large clusters that need longer to start or run crash recovery. */
+  /** Milliseconds to wait for PostgreSQL to accept connections before failing (default 60000). Also passed to `pg_ctl` as `-t`, bounding its own wait for startup and for the shutdown checkpoint. Raise for large clusters that need longer to start, run crash recovery, or shut down. */
   readyTimeout?: number
 }
 
@@ -193,7 +195,8 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
    * this uses `pg_dump` to create a logical dump before backup and `pg_restore` to rebuild
    * the database after restore.
    *
-   * The dump file is written directly to the backup target — no data duplication on disk.
+   * The dump is staged in the subcontainer's `/tmp` and copied to the backup
+   * target, so it needs transient room for one copy of the dump.
    *
    * @returns A configured Backups instance with pre/post hooks. Chain `.addVolume()` or
    * `.addSync()` to include additional volumes/paths in the backup.
@@ -214,6 +217,7 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
       readyTimeout = 60_000,
     } = config
     const pgdata = `${mountpoint}${pgdataPath}`
+    const pgCtlTimeout = String(Math.ceil(readyTimeout / 1000))
     const dumpFile = `${BACKUP_CONTAINER_MOUNT}/${database}-db.dump`
     // pg_dump's writes are silently dropped on the backup-fs FUSE mount —
     // pg_dump exits 0 but the file stays 0 bytes. `cp` writes through the
@@ -231,17 +235,7 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
       })
     }
 
-    async function startPg(
-      sub: {
-        exec(cmd: string[], opts?: any): Promise<{ exitCode: number | null }>
-        execFail(
-          cmd: string[],
-          opts?: any,
-          timeout?: number | null,
-        ): Promise<any>
-      },
-      label: string,
-    ) {
+    async function startPg(sub: SubContainer<M>, label: string) {
       await sub.exec(['rm', '-f', `${pgdata}/postmaster.pid`], {
         user: 'postgres',
       })
@@ -255,9 +249,20 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
       const pgStartOpts = pgOptions
         ? `-c listen_addresses= ${pgOptions}`
         : '-c listen_addresses='
-      await sub.execFail(['pg_ctl', 'start', '-D', pgdata, '-o', pgStartOpts], {
-        user: 'postgres',
-      })
+      await sub.execFail(
+        [
+          'pg_ctl',
+          'start',
+          '-D',
+          pgdata,
+          '-t',
+          pgCtlTimeout,
+          '-o',
+          pgStartOpts,
+        ],
+        { user: 'postgres' },
+        NO_TIMEOUT,
+      )
       for (let elapsed = 0; elapsed < readyTimeout; elapsed += 1000) {
         const { exitCode } = await sub.exec(['pg_isready', '-U', user], {
           user: 'postgres',
@@ -292,14 +297,20 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
             await sub.execFail(
               ['pg_dump', '-U', user, '-Fc', '-f', tmpDumpFile, database],
               { user: 'postgres' },
-              null,
+              NO_TIMEOUT,
             )
             console.log('[pg-dump] copying dump to backup target')
-            await sub.execFail(['cp', tmpDumpFile, dumpFile], { user: 'root' })
+            await sub.execFail(
+              ['cp', tmpDumpFile, dumpFile],
+              { user: 'root' },
+              NO_TIMEOUT,
+            )
             console.log('[pg-dump] stopping postgres')
-            await sub.execFail(['pg_ctl', 'stop', '-D', pgdata, '-w'], {
-              user: 'postgres',
-            })
+            await sub.execFail(
+              ['pg_ctl', 'stop', '-D', pgdata, '-w', '-t', pgCtlTimeout],
+              { user: 'postgres' },
+              NO_TIMEOUT,
+            )
             console.log('[pg-dump] complete')
           },
         )
@@ -315,17 +326,23 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
             await mountBackupTarget(sub.rootfs)
             // Stage the dump off the FUSE before pg_restore reads it — see
             // the comment on `tmpDumpFile` above.
-            await sub.execFail(['cp', dumpFile, tmpDumpFile], { user: 'root' })
+            await sub.execFail(
+              ['cp', dumpFile, tmpDumpFile],
+              { user: 'root' },
+              NO_TIMEOUT,
+            )
             await sub.execFail(['chown', 'postgres:postgres', tmpDumpFile], {
               user: 'root',
             })
             await sub.execFail(
               ['chown', '-R', 'postgres:postgres', mountpoint],
               { user: 'root' },
+              NO_TIMEOUT,
             )
             await sub.execFail(
               ['initdb', '-D', pgdata, '-U', user, ...initdbArgs],
               { user: 'postgres' },
+              NO_TIMEOUT,
             )
             await startPg(sub, 'pg-restore')
             await sub.execFail(['createdb', '-U', user, database], {
@@ -343,7 +360,7 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
                 tmpDumpFile,
               ],
               { user: 'postgres' },
-              null,
+              NO_TIMEOUT,
             )
             if (resolvedPassword !== null) {
               await sub.execFail(
@@ -359,9 +376,11 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
                 { user: 'postgres' },
               )
             }
-            await sub.execFail(['pg_ctl', 'stop', '-D', pgdata, '-w'], {
-              user: 'postgres',
-            })
+            await sub.execFail(
+              ['pg_ctl', 'stop', '-D', pgdata, '-w', '-t', pgCtlTimeout],
+              { user: 'postgres' },
+              NO_TIMEOUT,
+            )
           },
         )
       })
@@ -374,7 +393,8 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
    * this uses `mysqldump` to create a logical dump before backup and `mysql` to restore
    * the database after restore.
    *
-   * The dump file is stored temporarily in `dumpVolume` during backup and cleaned up afterward.
+   * The dump is staged in the subcontainer's `/tmp` and copied to the backup
+   * target, so it needs transient room for one copy of the dump.
    *
    * @returns A configured Backups instance with pre/post hooks. Chain `.addVolume()` or
    * `.addSync()` to include additional volumes/paths in the backup.
@@ -409,10 +429,7 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
       })
     }
 
-    async function waitForMysql(
-      sub: { exec(cmd: string[]): Promise<{ exitCode: number | null }> },
-      cmd: string[],
-    ) {
+    async function waitForMysql(sub: SubContainer<M>, cmd: string[]) {
       for (let elapsed = 0; elapsed < readyTimeout; elapsed += 1000) {
         const { exitCode } = await sub.exec(cmd)
         if (exitCode === 0) return
@@ -423,10 +440,7 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
       )
     }
 
-    async function startMysql(sub: {
-      exec(cmd: string[], opts?: any): Promise<{ exitCode: number | null }>
-      execFail(cmd: string[], opts?: any, timeout?: number | null): Promise<any>
-    }) {
+    async function startMysql(sub: SubContainer<M>) {
       if (engine === 'mariadb') {
         // MariaDB doesn't support --daemonize; fire-and-forget the exec
         sub
@@ -439,6 +453,7 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
               ...mysqldOptions,
             ],
             { user: 'root' },
+            NO_TIMEOUT,
           )
           .catch(e =>
             console.error('[mysql-backup] mysqld exited unexpectedly:', e),
@@ -454,15 +469,12 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
             ...mysqldOptions,
           ],
           { user: 'root' },
-          null,
+          NO_TIMEOUT,
         )
       }
     }
 
-    async function stopMysql(sub: {
-      exec(cmd: string[], opts?: any): Promise<{ exitCode: number | null }>
-      execFail(cmd: string[], opts?: any, timeout?: number | null): Promise<any>
-    }) {
+    async function stopMysql(sub: SubContainer<M>) {
       // SIGTERM mysqld and wait for it to finish flushing before teardown.
       // A killed-but-unreaped mysqld lingers as a zombie that keeps its PID,
       // so `tail --pid`/`kill -0` would block here forever — treat the zombie
@@ -485,7 +497,7 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
           ].join('\n'),
         ],
         { user: 'root' },
-        null,
+        NO_TIMEOUT,
       )
     }
 
@@ -514,9 +526,11 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
               user: 'root',
             })
             if (engine === 'mysql') {
-              await sub.execFail(['chown', '-R', 'mysql:mysql', datadir], {
-                user: 'root',
-              })
+              await sub.execFail(
+                ['chown', '-R', 'mysql:mysql', datadir],
+                { user: 'root' },
+                NO_TIMEOUT,
+              )
             }
             await startMysql(sub)
             await waitForMysql(sub, readyCmd)
@@ -531,9 +545,13 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
                 database,
               ],
               { user: 'root' },
-              null,
+              NO_TIMEOUT,
+            )
+            await sub.execFail(
+              ['cp', tmpDumpFile, dumpFile],
+              { user: 'root' },
+              NO_TIMEOUT,
             )
-            await sub.execFail(['cp', tmpDumpFile, dumpFile], { user: 'root' })
             await stopMysql(sub)
           },
         )
@@ -558,6 +576,7 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
               await sub.execFail(
                 ['mysql_install_db', '--user=mysql', `--datadir=${datadir}`],
                 { user: 'root' },
+                NO_TIMEOUT,
               )
             } else {
               await sub.execFail(
@@ -568,6 +587,7 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
                   `--datadir=${datadir}`,
                 ],
                 { user: 'root' },
+                NO_TIMEOUT,
               )
             }
             await startMysql(sub)
@@ -589,7 +609,11 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
             })
             // Stage the dump off the FUSE before mysql reads it — see
             // the comment on `tmpDumpFile` above.
-            await sub.execFail(['cp', dumpFile, tmpDumpFile], { user: 'root' })
+            await sub.execFail(
+              ['cp', dumpFile, tmpDumpFile],
+              { user: 'root' },
+              NO_TIMEOUT,
+            )
             // Restore from dump
             await sub.execFail(
               [
@@ -598,7 +622,7 @@ export class Backups<M extends T.SDKManifest> implements InitScript {
                 `mysql -u root ${pw !== null ? `-p'${pw}'` : ''} ${database} < ${tmpDumpFile}`,
               ],
               { user: 'root' },
-              null,
+              NO_TIMEOUT,
             )
             await stopMysql(sub)
           },