SCHISM on SkyPilot: Test_CORIE example

Runs the SCHISM ocean model's Test_CORIE example (Columbia River estuary, 28-day real-tide/atmospheric-forcing hindcast) on a single cloud VM via SkyPilot, building SCHISM from source with UGRID-compliant output, and uploads its own output to S3 as part of the run. A separate post-processing script turns that S3 output into a queryable virtual Icechunk store, explored in a notebook with xarray + xugrid + hvplot.

One-time setup

pip install "skypilot[aws,gcp]"
sky check                                        # confirm at least one cloud is enabled

aws s3 mb s3://<your-bucket-name> --region us-east-1
./scripts/setup-runner-role.sh <your-bucket-name>  # persistent, scoped IAM role -- see below

If you use a bucket name other than the default baked into schism-corie.yaml (envs.BUCKET), update that value too.

Quick start

sky launch --dryrun schism-corie.yaml            # sanity-check + cost estimate

# Smoke test: shortened 1-day run to validate the whole pipeline first
sky launch -y -c schism-corie schism-corie.yaml --env RNDAY_OVERRIDE=1

# Full 28-day run (reuses the cached build/data via sky exec)
sky exec schism-corie schism-corie.yaml --env RNDAY_OVERRIDE=28

# Output is already in S3 by the time the run above finishes (see "What
# this does"). Tear down when done -- though the cluster also auto-tears
# itself down after 30 idle minutes regardless (see below).
sky down schism-corie

# Post-processing (run locally, not on the cluster): turn that run's S3
# output into a virtual Icechunk store -- no data copied, just byte-range
# references. Needs its own deps: pip install virtualizarr icechunk obstore
# obspec-utils xarray boto3
AWS_PROFILE=<profile> python3 scripts/build_virtual_icechunk.py \
    --bucket schism-corie-output-097532040392 --tag <RUN_TAG from the sync log>

# Then explore it: notebooks/explore_icechunk.ipynb (xarray + xugrid + hvplot)

What this does

• Provisions a single VM (cpus: 40+, sized to match the reference run's 40 MPI ranks) on whichever of AWS/GCP is enabled and cheapest. Runs directly on the VM, not inside a container -- there's no isolation benefit to gain from containerizing a single-tenant HPC build, and it only adds a network hop that complicates instance-metadata (and thus credential) access (see the retired Docker-based approach, git history, for what that cost us).
• ~/.sky/config.yaml's aws.remote_identity (pattern-matched to the schism-corie cluster name) attaches a dedicated, narrowly-scoped IAM role (S3 write to exactly one bucket, nothing else) instead of SkyPilot's own shared default role -- see "One-time setup".
• resources.autostop tears the cluster down automatically after 30 idle minutes (no running job, no SSH session). This exists because a real run of this pipeline once sat idle for ~49 hours after finishing (~$99 in unnecessary compute) simply because nobody remembered sky down.
• setup: installs build deps via apt, clones schism-dev/schism, builds pschism with CMake using scribed I/O (-DOLDIO=OFF), and pulls the Test_CORIE test case data via svn export.
• run: inserts iof_ugrid = 2 into param.nml (see "UGRID output" below), copies the built executable, optionally shortens rnday for a smoke test, runs mpirun ... ./pschism <NSCRIBES>, and then syncs outputs/ to a per-run S3 prefix (timestamp + rnday, e.g. outputs/20260716T012233Z-rnday1/) using the instance's own role -- no separate upload step needed, and no risk of one run's sync overwriting another's (see the incident note under "The IAM role" below). Skips re-running mpirun if outputs/mirror.out already shows a completed run, so re-invoking (e.g. after an interrupted sync) doesn't redo the compute.

Non-obvious fixes baked into schism-corie.yaml

These were found by actually running the pipeline, not just reading docs:

1. -DBLD_STANDALONE=ON -DBUILD_TOOLS=ON must be passed explicitly. SCHISM's own src/CMakeLists.txt calls its define_opt(name, doc, default) macro with arguments in the wrong order for these two flags (define_opt(BLD_STANDALONE "Standalone build mode" ON) — doc and default are swapped relative to the macro's (name, default, doc) signature). The result: both options silently cache as OFF, the Driver/ subdirectory never gets configured, and make pschism fails with No rule to make target 'pschism'. Overriding both on the command line sidesteps the bug.
2. SCHISM's build tags the executable's filename with the enabled options, e.g. pschism_OLDIO_BLD_STANDALONE_BUILD_TOOLS_TVD-SB — the CMake target is pschism, but the binary is not. run: copies via pschism_* glob rather than a literal name.
3. columbia.vims.edu's TLS cert is expired server-side (not a local CA-trust issue). svn export needs --trust-server-cert-failures=unknown-ca,expired,cn-mismatch,other.
4. mpirun needs --use-hwthread-cpus on 2-threads-per-core cloud instances (e.g. c6i.12xlarge = 24 physical cores / 48 vCPUs) — OpenMPI's default slot count is physical cores, so -np 40 exceeds it without this flag. Unrelated to the container/VM question above -- this is purely about vCPU vs. physical core counts on the instance itself.
5. SCHISM writes its own log to outputs/mirror.out, not stdout — check that file (not the run: block's captured stdout) for the Run completed successfully message.
6. Scribed I/O (OLDIO=OFF) needs an explicit scribe-count argument: mpirun -np NPROC ./pschism NSCRIBES. NSCRIBES must be at least (# enabled 3D outputs, with vector fields counting as 2) + 1 for the 2D bundle -- too few and the run errors out; extra scribes just sit idle. envs.NSCRIBES in schism-corie.yaml is set for Test_CORIE's default iof_hydro selections; changing which variables are enabled means recomputing this.
7. Test_CORIE's shipped param.nml predates the iof_ugrid flag entirely -- it's not in the &SCHOUT namelist at all, commented or otherwise, so run: appends the line rather than trying to toggle an existing one.
8. A bare ! in a double-quoted sed pattern triggers bash history expansion under SkyPilot's task runner (event not found), even though the script isn't interactive. Escape it as \!.

UGRID output and the virtual Icechunk store

iof_ugrid is a param.nml flag, only meaningful under scribed I/O (OLDIO=OFF): 0 disables UGRID metadata, 1 embeds full mesh data in every 3D output file, 2 (what this repo uses) has 3D files reference the mesh data already written once in out2d_*.nc. With it set, out2d_*.nc carries a proper cf_role="mesh_topology" variable, face/edge connectivity, and Conventions: CF-1.12 -- verified directly against the actual output, not assumed from the flag's docstring.

scripts/build_virtual_icechunk.py (run locally, not on the cluster -- see Quick start) groups a run's per-variable NetCDF files (out2d_*.nc, temperature_*.nc, salinity_*.nc, etc. -- one file per variable per output stack), virtualizes each group with VirtualiZarr (byte-range references, no data copied out of S3), merges all variables into one dataset, and commits it to an Icechunk repo at s3://<bucket>/icechunk/<tag>/. Two things worth knowing if you touch this script:

• xr.concat's default data_vars="all" stacks every data variable along the concat dimension, including static grid/connectivity arrays that don't vary between stacks and have no time dimension of their own -- giving them a spurious extra dimension that xugrid then rejects. Fixed with data_vars="minimal".
• Neither obstore nor icechunk read AWS_PROFILE or a shared credentials file, and there's no EC2 instance-metadata endpoint to fall back to when running this locally (as opposed to on the cluster, where the scoped instance role Just Works). Credentials are resolved via boto3.Session and passed through explicitly.

notebooks/explore_icechunk.ipynb opens that Icechunk store with xr.open_zarr + xu.UgridDataset, and plots sample maps with hvplot.xugrid. One extra step in there: Test_CORIE's hgrid.gr3 is authored in Oregon State Plane North, NAD27, in meters (per its own header) -- there's no separately-registered EPSG code for that exact zone/datum in meters (the standard code, EPSG:32026, is in US survey feet). Since PROJ's false easting/northing are always in meters regardless of +units, swapping +units=us-ft -> +units=m in EPSG:32026's proj4 string gives the same physical projection while matching the header. The notebook uses xugrid's Ugrid2d.set_crs/.ugrid.to_crs() to actually reproject the grid topology (not just relabel the coordinate arrays) to lon/lat, which is what makes geo=True/basemap tiles work in the plots.

Public copy for browser viewers (gridlook)

scripts/publish_public_icechunk.py --tag <TAG> (run with a profile that can write to the target bucket) copies the run's NetCDFs to s3://esip-qhub-public/rsignell/schism-corie/<TAG>/ and builds an anonymous virtual Icechunk repo at .../<TAG>.icechunk (bucket is us-west-2). Versus the private store it adds real SCHISM_hgrid_{node,edge,face}_{lon,lat} variables (same Oregon North reprojection as above) and a (nface, 3) SCHISM_hgrid_face_triangles array, and repoints the SCHISM_hgrid mesh attributes at them -- gridlook needs degrees and 3-column triangle connectivity. The bucket's CORS must allow gridlook's origin. gridlook link:

https://feat-ugrid.gridlook.pages.dev/#https://esip-qhub-public.s3.amazonaws.com/rsignell/schism-corie/20260717T143705Z-rnday3.icechunk::varname=temperature::dimIndices_time=0::dimIndices_nSCHISM_vgrid_layers=53

The IAM role (scripts/setup-runner-role.sh / teardown-runner-role.sh)

setup-runner-role.sh <bucket-name> is a one-time, idempotent step: it creates a persistent schism-corie-runner IAM role (trust: ec2.amazonaws.com; permissions: s3:PutObject/s3:ListBucket on exactly the bucket you name, nothing else) and an instance profile wrapping it, and adds a schism-corie: schism-corie-runner entry to ~/.sky/config.yaml's aws.remote_identity list (pattern-matched by cluster name, so it only affects clusters named schism-corie -- everything else keeps SkyPilot's normal default role). This attaches the role to every launch automatically -- no per-run credential setup, no swapping SkyPilot's own instance profile back and forth.

teardown-runner-role.sh deletes the role, if you're done with this project entirely. Day-to-day, you don't need it -- just sky down.

A gotcha worth knowing regardless of the above: SkyPilot's default remote_identity behavior on AWS (when nothing overrides it) uploads your entire local ~/.aws/credentials file -- every profile configured on your machine, not just the active one -- to the cluster. Those static keys are checked before any instance role in AWS's default credential chain -- if a [default] profile with stale/invalid keys happens to exist there, it silently shadows a perfectly good instance role, and every aws call on the cluster fails with InvalidClientTokenId, which looks like a role/permissions problem but isn't one. Naming a role explicitly via remote_identity (as this repo does) avoids that upload entirely -- confirmed empirically: ssh schism-corie "ls ~/.aws/" shows no such directory at all with this setup, and aws sts get-caller-identity on the cluster correctly reports the assumed schism-corie-runner role.

An incident this design fixes: an earlier version of this pipeline synced every run to the same fixed outputs/ S3 prefix. Running a 1-day smoke test after a real 28-day run had already been uploaded silently overwrote that 28-day run's results -- same rank count means identical filenames, and aws s3 sync has no collision protection and nothing was using bucket versioning. The real 28-day output was unrecoverable. run: now writes to a unique, timestamped prefix per invocation specifically to make that impossible going forward.

Validation

A full 28-day run (rnday=28, dt=90, 26,880 steps) completed successfully on AWS c6i.12xlarge in ~3.7 hours, matching the reference log (mirror.out.ref) step count, with an empty fatal.error and physically sane tidal elevations (final average |eta| ≈ 1.57 m). Total compute cost for that validated pipeline (smoke test + full run) was under $15 -- a separate, unrelated ~49-hour idle stretch afterward (now prevented by resources.autostop) cost roughly $99 more and is not part of that figure. Output (~59GB, OLDIO per-rank NetCDF + hotstart files) is not committed to this repo; it lives in S3.

Note: that 28-day validation predates the switch to scribed I/O (OLDIO=OFF) and UGRID output -- it was run under the now-retired OLDIO=ON configuration. The current OLDIO=OFF/iof_ugrid=2 configuration has only been validated at 3-day scale (rnday=3, 2,880 steps) so far: run completed successfully, out2d_1.nc confirmed UGRID-compliant by direct inspection (cf_role="mesh_topology", face/edge connectivity, Conventions: CF-1.12), and the resulting Icechunk store round-tripped real data correctly (checked actual elevation/salinity values through the virtual references, not just structure). It has not yet been re-run at the full 28-day scale.