Docker 18.09 fails to create containers when MountFlags=slave is set

[haxorof]

Expected behavior

[root@localhost ~]# docker run -it --rm hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
d1725b59e92d: Pull complete
Digest: sha256:b3a26e22bf55e4a5232b391281fc1673f18462b75cdc76aa103e6d3a2bce5e77
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Actual behavior

[root@localhost ~]# docker run -it --rm hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
d1725b59e92d: Pull complete
Digest: sha256:b3a26e22bf55e4a5232b391281fc1673f18462b75cdc76aa103e6d3a2bce5e77
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/hello\": stat /hello: no such file or directory": unknown.

Steps to reproduce the behavior

- name: Good case
  hosts: test-host
  vars:
    docker_enable_mount_flag_fix: no
  roles:
    - haxorof.docker-ce
  post_tasks:
    - name: Test hello container
      become: yes
      shell: docker run --rm hello-world

- name: Bad case
  hosts: test-host
  roles:
    - haxorof.docker-ce
  post_tasks:
    - name: Test hello container
      become: yes
      shell: docker run --rm hello-world

Output of cat /etc/docker/daemon.json:

{
  "debug": true
}

Output of docker version:

Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:48:22 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:19:08 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 18.09.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-957.1.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.605GiB
Name: localhost.localdomain
ID: 4IYK:IGY7:J4XB:XXX4:7F25:52LF:VUGS:RWYQ:B6T5:OZCH:6LKU:ZBBF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 22
 Goroutines: 38
 System Time: 2018-12-27T16:55:07.914762765Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: bridge-nf-call-ip6tables is disabled

Output of journalctl -u docker -n 50 -l --no-pager:

level=info msg="API listen on /var/run/docker.sock"
level=debug msg="Calling GET /_ping"
level=debug msg="Calling POST /v1.39/containers/create"
level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":true,\"AttachStdout\":true,\"Cmd\":null,\"Domainname\":\"\",\"Entrypoint\":null,\"Env\":[],\"HostConfig\":{\"AutoRemove\":true,\"Binds\":null,\"BlkioDeviceReadBps\":null,\"BlkioDeviceReadIOps\":null,\"BlkioDeviceWriteBps\":null,\"BlkioDeviceWriteIOps\":null,\"BlkioWeight\":0,\"BlkioWeightDevice\":[],\"CapAdd\":null,\"CapDrop\":null,\"Cgroup\":\"\",\"CgroupParent\":\"\",\"ConsoleSize\":[0,0],\"ContainerIDFile\":\"\",\"CpuCount\":0,\"CpuPercent\":0,\"CpuPeriod\":0,\"CpuQuota\":0,\"CpuRealtimePeriod\":0,\"CpuRealtimeRuntime\":0,\"CpuShares\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"DeviceCgroupRules\":null,\"Devices\":[],\"DiskQuota\":0,\"Dns\":[],\"DnsOptions\":[],\"DnsSearch\":[],\"ExtraHosts\":null,\"GroupAdd\":null,\"IOMaximumBandwidth\":0,\"IOMaximumIOps\":0,\"IpcMode\":\"\",\"Isolation\":\"\",\"KernelMemory\":0,\"Links\":null,\"LogConfig\":{\"Config\":{},\"Type\":\"\"},\"MaskedPaths\":null,\"Memory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwappiness\":-1,\"NanoCpus\":0,\"NetworkMode\":\"default\",\"OomKillDisable\":false,\"OomScoreAdj\":0,\"PidMode\":\"\",\"PidsLimit\":0,\"PortBindings\":{},\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyPaths\":null,\"ReadonlyRootfs\":false,\"RestartPolicy\":{\"MaximumRetryCount\":0,\"Name\":\"no\"},\"SecurityOpt\":null,\"ShmSize\":0,\"UTSMode\":\"\",\"Ulimits\":null,\"UsernsMode\":\"\",\"VolumeDriver\":\"\",\"VolumesFrom\":null},\"Hostname\":\"\",\"Image\":\"hello-world\",\"Labels\":{},\"NetworkingConfig\":{\"EndpointsConfig\":{}},\"OnBuild\":null,\"OpenStdin\":true,\"StdinOnce\":true,\"Tty\":true,\"User\":\"\",\"Volumes\":{},\"WorkingDir\":\"\"}"
level=debug msg="Calling GET /v1.39/info"
level=debug msg="Calling POST /v1.39/images/create?fromImage=hello-world&tag=latest"
level=debug msg="Trying to pull hello-world from https://registry-1.docker.io v2"
level=debug msg="Pulling ref from V2 registry: hello-world:latest"
level=debug msg="docker.io/library/hello-world:latest resolved to a manifestList object with 10 entries; looking for a unknown/amd64 match"
level=debug msg="found match for linux/amd64 with media type application/vnd.docker.distribution.manifest.v2+json, digest sha256:1a6fd470b9ce10849be79e99529a88371dff60c60aab424c077007f6979b4812"
level=debug msg="pulling blob \"sha256:d1725b59e92d6462c6d688ef028979cc6bb150762db99d18dddc7fa54b82b0ce\""
level=debug msg="Downloaded d1725b59e92d to tempfile /var/lib/docker/tmp/GetImageBlob714643933"
level=debug msg="Applying tar in /var/lib/docker/overlay2/54098eb647e241bba97e4106e9b8bd60a67ffcff71127a83bc4bed9b7e2b4f13/diff" storage-driver=overlay2
level=debug msg="Applied tar sha256:428c97da766c4c13b19088a471de6b622b038f3ae8efa10ec5a37d6d31a2df0b to 54098eb647e241bba97e4106e9b8bd60a67ffcff71127a83bc4bed9b7e2b4f13, size: 1840"
level=debug msg="Calling POST /v1.39/containers/create"
level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":true,\"AttachStdout\":true,\"Cmd\":null,\"Domainname\":\"\",\"Entrypoint\":null,\"Env\":[],\"HostConfig\":{\"AutoRemove\":true,\"Binds\":null,\"BlkioDeviceReadBps\":null,\"BlkioDeviceReadIOps\":null,\"BlkioDeviceWriteBps\":null,\"BlkioDeviceWriteIOps\":null,\"BlkioWeight\":0,\"BlkioWeightDevice\":[],\"CapAdd\":null,\"CapDrop\":null,\"Cgroup\":\"\",\"CgroupParent\":\"\",\"ConsoleSize\":[0,0],\"ContainerIDFile\":\"\",\"CpuCount\":0,\"CpuPercent\":0,\"CpuPeriod\":0,\"CpuQuota\":0,\"CpuRealtimePeriod\":0,\"CpuRealtimeRuntime\":0,\"CpuShares\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"DeviceCgroupRules\":null,\"Devices\":[],\"DiskQuota\":0,\"Dns\":[],\"DnsOptions\":[],\"DnsSearch\":[],\"ExtraHosts\":null,\"GroupAdd\":null,\"IOMaximumBandwidth\":0,\"IOMaximumIOps\":0,\"IpcMode\":\"\",\"Isolation\":\"\",\"KernelMemory\":0,\"Links\":null,\"LogConfig\":{\"Config\":{},\"Type\":\"\"},\"MaskedPaths\":null,\"Memory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwappiness\":-1,\"NanoCpus\":0,\"NetworkMode\":\"default\",\"OomKillDisable\":false,\"OomScoreAdj\":0,\"PidMode\":\"\",\"PidsLimit\":0,\"PortBindings\":{},\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyPaths\":null,\"ReadonlyRootfs\":false,\"RestartPolicy\":{\"MaximumRetryCount\":0,\"Name\":\"no\"},\"SecurityOpt\":null,\"ShmSize\":0,\"UTSMode\":\"\",\"Ulimits\":null,\"UsernsMode\":\"\",\"VolumeDriver\":\"\",\"VolumesFrom\":null},\"Hostname\":\"\",\"Image\":\"hello-world\",\"Labels\":{},\"NetworkingConfig\":{\"EndpointsConfig\":{}},\"OnBuild\":null,\"OpenStdin\":true,\"StdinOnce\":true,\"Tty\":true,\"User\":\"\",\"Volumes\":{},\"WorkingDir\":\"\"}"
level=debug msg="container mounted via layerStore: &{/var/lib/docker/overlay2/08befbf7baf757b02a9a674a00ee4be03a0468d84ce55a3562d1cec80043aec4/merged 0x560031cb98a0 0x560031cb98a0}"
level=debug msg="Calling POST /v1.39/containers/cc7dec85bc4d4aad2d1e8fbb269d43347b44732f16bcc377309474c6f17cdad1/attach?stderr=1&stdin=1&stdout=1&stream=1"
level=debug msg="attach: stdin: begin"
level=debug msg="attach: stdout: begin"
level=debug msg="attach: stderr: begin"
level=debug msg="Calling POST /v1.39/containers/cc7dec85bc4d4aad2d1e8fbb269d43347b44732f16bcc377309474c6f17cdad1/wait?condition=removed"
level=debug msg="Calling POST /v1.39/containers/cc7dec85bc4d4aad2d1e8fbb269d43347b44732f16bcc377309474c6f17cdad1/start"
level=debug msg="container mounted via layerStore: &{/var/lib/docker/overlay2/08befbf7baf757b02a9a674a00ee4be03a0468d84ce55a3562d1cec80043aec4/merged 0x560031cb98a0 0x560031cb98a0}"
level=debug msg="Assigning addresses for endpoint confident_lichterman's interface on network bridge"
level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[])"
level=debug msg="Request address PoolID:172.17.0.0/16 App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0 Serial:false PrefAddress:<nil> "
level=debug msg="Assigning addresses for endpoint confident_lichterman's interface on network bridge"
level=debug msg="Programming external connectivity on endpoint confident_lichterman (c531a3758953ddbb0096cdeeea3b9e1c8b470bfc62b0196e1e210787dd62d64b)"
level=debug msg="EnableService cc7dec85bc4d4aad2d1e8fbb269d43347b44732f16bcc377309474c6f17cdad1 START"
level=debug msg="EnableService cc7dec85bc4d4aad2d1e8fbb269d43347b44732f16bcc377309474c6f17cdad1 DONE"
level=debug msg="bundle dir created" bundle=/var/run/docker/containerd/cc7dec85bc4d4aad2d1e8fbb269d43347b44732f16bcc377309474c6f17cdad1 module=libcontainerd namespace=moby root=/var/lib/docker/overlay2/08befbf7baf757b02a9a674a00ee4be03a0468d84ce55a3562d1cec80043aec4/merged
level=debug msg="sandbox set key processing took 61.306788ms for container cc7dec85bc4d4aad2d1e8fbb269d43347b44732f16bcc377309474c6f17cdad1"
level=error msg="stream copy error: reading from a closed fifo"
level=debug msg="attach: stdout: end"
level=debug msg="attach: stdin: end"
level=debug msg="attach: stderr: end"
level=debug msg="attach done"
level=debug msg="Closing buffered stdin pipe"
level=debug msg="Revoking external connectivity on endpoint confident_lichterman (c531a3758953ddbb0096cdeeea3b9e1c8b470bfc62b0196e1e210787dd62d64b)"
level=debug msg="DeleteConntrackEntries purged ipv4:0, ipv6:0"
level=debug msg="Releasing addresses for endpoint confident_lichterman's interface on network bridge"
level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)"
level=debug msg="Released address PoolID:LocalDefault/172.17.0.0/16, Address:172.17.0.2 Sequence:App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65532, Sequence: (0xe0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:3"
level=debug msg="Failed to remove 08befbf7baf757b02a9a674a00ee4be03a0468d84ce55a3562d1cec80043aec4 overlay: directory not empty" storage-driver=overlay2
level=error msg="cc7dec85bc4d4aad2d1e8fbb269d43347b44732f16bcc377309474c6f17cdad1 cleanup: failed to delete container from containerd: no such container"

Additional environment details (AWS, VirtualBox, physical, etc.)

Tested this using VirtualBox 6.0 and using Vagrant box centos/7 (https://app.vagrantup.com/centos/boxes/7)

Output of cat /etc/redhat-release:

CentOS Linux release 7.6.1810 (Core)

Output of uname -a:

Linux localhost.localdomain 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux