Docker named volumes don't share data across containers in a dstack CVM

[h4x3rotab]

Platform: dstack-dev-0.5.9 on prod5 (US-WEST-1) via Phala Cloud
Docker: the version shipped in the dstack OS image
Use case: init container writes derived state to a tmpfs/shared
volume; sibling containers read it on startup. Standard
docker-compose pattern.

Summary

Docker named volumes declared in docker-compose.yaml and mounted
into multiple containers behave as per-container ephemeral
storage rather than shared storage. Writes from container A are
not visible to container B even though both have the same volume
mounted at the same path. The host's
/var/lib/docker/volumes/<name>/_data/ shows up empty even right
after a container has supposedly written to it.

Reproduce on a fresh CVM

After SSH'ing into any deployed dstack CVM:

# Create a fresh named volume.
docker volume create probe

# Container A writes a file via the volume mount.
docker run --rm -v probe:/x alpine sh -c 'echo bind-test > /x/y.txt; ls -la /x'
# Output:
#   total 5
#   drwxr-xr-x    2 root     root            60 May  2 07:36 .
#   drwxr-xr-x    1 root     root             3 May  2 07:36 ..
#   -rw-r--r--    1 root     root            10 May  2 07:36 y.txt

# Container B reads the same volume — file is missing.
docker run --rm -v probe:/x alpine cat /x/y.txt
# Output:
#   cat: can't open '/x/y.txt': No such file or directory

# Host filesystem confirms the volume is empty:
ls -la /var/lib/docker/volumes/probe/_data/
# Output:
#   total 1
#   drwxr-xr-x    2 root     root             2 May  2 07:11 .
#   drwx-----x    3 root     root             4 May  2 07:11 ..

For comparison, bind mounts to /tmp work as expected:

mkdir -p /tmp/probe
docker run --rm -v /tmp/probe:/x alpine sh -c 'echo bind-test > /x/y.txt'
docker run --rm -v /tmp/probe:/x alpine cat /x/y.txt
# Output: bind-test

So the issue is specific to docker-named-volumes; bind mounts work.

Why this matters

Docker-compose's default and most-recommended way of sharing state
between sibling containers is via named volumes:

volumes:
  shared:

services:
  init:
    image: my-init
    volumes:
      - shared:/state          # write here
  app:
    image: my-app
    volumes:
      - shared:/state:ro       # read here
    depends_on:
      init:
        condition: service_completed_successfully

This pattern is broken on dstack today. Anyone running an init
container that materialises secrets / config / certs into a shared
volume hits this. Workaround is bind-mounting host paths under
/tmp/, but it's surprising and undocumented.

Where I hit this

Building a stage of an experimental Consul service mesh across
dstack CVMs (mesh-conn overlay, see related issues #242 / #243).
The init container uses the dstack Go SDK to derive cluster-wide
secrets via getKey(), write them to /run/secrets/<name>, and
exit; sibling consul/mesh-conn/coturn read them on startup. The
init container worked, said "wrote /run/secrets/gossip", exited 0
— and every sibling reported the file missing.

Lots of debugging time spent assuming the volume-share was
straightforward. Worth at minimum a docs note ("docker named
volumes don't share on dstack CVMs; use bind mounts to /tmp/ for
inter-container shared state") even if a real fix is not on the
roadmap.

Possible cause (speculation)

Could be the storage driver's behaviour in the dstack OS image
(maybe overlay2 with some option, or a noop driver), or a security
feature that isolates each container's view of /var/lib/docker/.
The host's view of the volume directory being EMPTY (not even
showing the file the container wrote) suggests the write went into
a copy-on-write layer that doesn't backflow to the named-volume
backing store. I don't know the dstack OS internals well enough to
say more.

Related

• Phala-Network/phala-cloud#242 — phala cvms list collapses replicas.
• Phala-Network/phala-cloud#243 — Per-instance Terraform resource + update_policy + lifecycle hooks.
• Phala-Network/terraform-provider-phala#5 — storage_fs ForceNew bug.

All four surfaced from the same Consul-on-dstack experiment.

[h4x3rotab]

Update: I cannot reproduce this on the same CVM after a clean apply. The issue I observed earlier was almost certainly a false positive — likely caused by interference with a live docker-compose stack that was actively mounting/remounting the same named volumes during my test, or by my own confusion with the volume's tmpfs driver_opts setting in an earlier compose iteration.

Re-running the canonical test shows named volumes share correctly:

$ docker volume create retest-shared
$ docker run --rm -v retest-shared:/x alpine sh -c 'echo VAL > /x/f.txt && ls -la /x'
total 2
-rw-r--r--    1 root     root             4 May  2 07:58 f.txt
$ ls -la /var/lib/docker/volumes/retest-shared/_data/
-rw-r--r--    1 root     root             4 May  2 07:58 f.txt
$ docker run --rm -v retest-shared:/x alpine cat /x/f.txt
VAL

Sharing works as expected.

Closing as not-a-bug. Sorry for the noise. Leaving the comment so the next person finding the issue via search has the resolution. Thanks @h4xrotab for catching this.