From 7aaee9c7d66c38373b3077f0b58bea8efa9dcfcf Mon Sep 17 00:00:00 2001 From: Alban Crequy Date: Tue, 20 Mar 2018 11:12:04 +0100 Subject: [PATCH 1/6] docs: update link to spec runtime-spec v1.0.1 is released. No need to link to a pre-v1 rc anymore. Signed-off-by: Alban Crequy --- docs/runtime-compliance-testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/runtime-compliance-testing.md b/docs/runtime-compliance-testing.md index edcca35c3..2ce4b1da9 100644 --- a/docs/runtime-compliance-testing.md +++ b/docs/runtime-compliance-testing.md @@ -6,4 +6,4 @@ In order to be tested for [compliance][], runtimes MUST support at least one of * Version 1.0.1 of the [OCI Runtime Command Line Interface](command-line-interface.md). -[compliance]: https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc4/spec.md#notational-conventions +[compliance]: https://github.com/opencontainers/runtime-spec/blob/v1.0.1/spec.md From 294ca2a32459d34228156fccb8c91f42822d2b95 Mon Sep 17 00:00:00 2001 From: Alban Crequy Date: Tue, 20 Mar 2018 12:32:45 +0100 Subject: [PATCH 2/6] docs: move compliance testing to separate doc Signed-off-by: Alban Crequy --- README.md | 90 +--------------- docs/runtime-compliance-testing.md | 158 +++++++++++++++++++++++++++++ 2 files changed, 159 insertions(+), 89 deletions(-) diff --git a/README.md b/README.md index 11f613cf9..ffbbdd0e2 100644 --- a/README.md +++ b/README.md @@ -30,99 +30,11 @@ INFO[0000] Bundle validation succeeded. ## Testing OCI runtimes -The runtime validation suite uses [node-tap][], which is packaged for some distributions (for example, it is in [Debian's `node-tap` package][debian-node-tap]). -If your distribution does not package node-tap, you can install [npm][] (for example, from [Gentoo's `nodejs` package][gentoo-nodejs]) and use it: - -```console -$ npm install tap -``` - -Build the validation executables: - -```console -$ make runtimetest validation-executables -``` - -Runtime validation currently [only supports](docs/runtime-compliance-testing.md) the [OCI Runtime Command Line Interface](docs/command-line-interface.md). -If we add support for alternative APIs in the future, runtime validation will gain an option to select the desired runtime API. -For the command line interface, the `RUNTIME` option selects the runtime command (`funC` in the [OCI Runtime Command Line Interface](docs/command-line-interface.md)). - -``` -$ sudo make RUNTIME=runc localvalidation -RUNTIME=runc tap validation/pidfile.t validation/linux_cgroups_hugetlb.t validation/linux_cgroups_memory.t validation/linux_rootfs_propagation_shared.t validation/kill.t validation/create.t validation/poststart.t validation/linux_cgroups_network.t validation/poststop_fail.t validation/linux_readonly_paths.t validation/prestart_fail.t validation/hooks_stdin.t validation/default.t validation/linux_masked_paths.t validation/poststop.t validation/misc_props.t validation/prestart.t validation/poststart_fail.t validation/mounts.t validation/linux_cgroups_relative_pids.t validation/process_user.t validation/process.t validation/hooks.t validation/process_capabilities_fail.t validation/process_rlimits_fail.t validation/linux_cgroups_relative_cpus.t validation/process_rlimits.t validation/linux_cgroups_relative_blkio.t validation/linux_sysctl.t validation/linux_seccomp.t validation/linux_devices.t validation/start.t validation/linux_cgroups_pids.t validation/process_capabilities.t validation/process_oom_score_adj.t validation/linux_cgroups_relative_hugetlb.t validation/linux_cgroups_cpus.t validation/linux_cgroups_relative_memory.t validation/state.t validation/root_readonly_true.t validation/linux_cgroups_blkio.t validation/linux_rootfs_propagation_unbindable.t validation/delete.t validation/linux_cgroups_relative_network.t validation/hostname.t validation/killsig.t validation/linux_uid_mappings.t -validation/pidfile.t .failed to create the container -container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 56892210544640 to hugetlb.1GB.limit_in_bytes: open /sys/fs/cgroup/hugetlb/cgrouptest/hugetlb.1GB.limit_in_bytes: permission denied\\\"\"" -exit status 1 -validation/pidfile.t .................................. 1/1 315ms -validation/linux_cgroups_hugetlb.t .................... 0/1 - not ok validation/linux_cgroups_hugetlb.t - timeout: 30000 - file: validation/linux_cgroups_hugetlb.t - command: validation/linux_cgroups_hugetlb.t - args: [] - stdio: - - 0 - - pipe - - 2 - cwd: /…/go/src/github.com/opencontainers/runtime-tools - exitCode: 1 - -validation/linux_cgroups_memory.t ..................... 9/9 -validation/linux_rootfs_propagation_shared.t ...... 252/282 - not ok shared root propagation exposes "/target348456609/mount892511628/example376408222" - - Skipped: 29 - /dev/null (default device) has unconfigured permissions -… -total ........................................... 4381/4962 - - - 4381 passing (1m) - 567 pending - 14 failing - -make: *** [Makefile:44: localvalidation] Error 1 -``` - -You can also run an individual test executable directly: - -```console -$ RUNTIME=runc validation/default.t -TAP version 13 -ok 1 - has expected hostname - --- - { - "actual": "mrsdalloway", - "expected": "mrsdalloway" - } - ... -… -ok 287 # SKIP linux.gidMappings not set -1..287 -``` - -If you cannot install node-tap, you can probably run the test suite with another [TAP consumer][tap-consumers]. -For example, with [`prove`][prove]: - -```console -$ sudo make TAP='prove -Q -j9' RUNTIME=runc VALIDATION_TESTS=validation/pidfile.t localvalidation -RUNTIME=runc prove -Q -j9 validation/pidfile.t -All tests successful. -Files=1, Tests=1, 0 wallclock secs ( 0.01 usr 0.01 sys + 0.03 cusr 0.03 csys = 0.08 CPU) -Result: PASS -``` +Use the [runtime validation suite](doc/runtime-compliance-testing.md). [bundle]: https://github.com/opencontainers/runtime-spec/blob/master/bundle.md [config.json]: https://github.com/opencontainers/runtime-spec/blob/master/config.md -[debian-node-tap]: https://packages.debian.org/stretch/node-tap -[debian-nodejs]: https://packages.debian.org/stretch/nodejs -[gentoo-nodejs]: https://packages.gentoo.org/packages/net-libs/nodejs -[node-tap]: http://www.node-tap.org/ -[npm]: https://www.npmjs.com/ -[prove]: http://search.cpan.org/~leont/Test-Harness-3.39/bin/prove [runC]: https://github.com/opencontainers/runc [runtime-spec]: https://github.com/opencontainers/runtime-spec -[tap-consumers]: https://testanything.org/consumers.html - [generate.1]: man/oci-runtime-tool-generate.1.md [validate.1]: man/oci-runtime-tool-validate.1.md diff --git a/docs/runtime-compliance-testing.md b/docs/runtime-compliance-testing.md index 2ce4b1da9..d65dd3799 100644 --- a/docs/runtime-compliance-testing.md +++ b/docs/runtime-compliance-testing.md @@ -6,4 +6,162 @@ In order to be tested for [compliance][], runtimes MUST support at least one of * Version 1.0.1 of the [OCI Runtime Command Line Interface](command-line-interface.md). +## Running the runtime validation suite from source + +The runtime validation suite uses [node-tap][], which is packaged for some distributions (for example, it is in [Debian's `node-tap` package][debian-node-tap]). +If your distribution does not package node-tap, you can install [npm][] (for example, from [Gentoo's `nodejs` package][gentoo-nodejs]) and use it: + +```console +$ npm install tap +``` + +Build the validation executables: + +```console +$ make runtimetest validation-executables +``` + +Runtime validation currently [only supports](docs/runtime-compliance-testing.md) the [OCI Runtime Command Line Interface](doc/command-line-interface.md). +If we add support for alternative APIs in the future, runtime validation will gain an option to select the desired runtime API. +For the command line interface, the `RUNTIME` option selects the runtime command (`funC` in the [OCI Runtime Command Line Interface](doc/command-line-interface.md)). + +``` +$ sudo make RUNTIME=runc localvalidation +RUNTIME=runc tap validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t +validation/linux_rootfs_propagation_shared.t ........ 18/19 + not ok rootfs propagation + error: 'rootfs should be shared, but not' + +validation/create.t ................................... 4/4 +validation/default.t ................................ 19/19 +validation/linux_readonly_paths.t ................... 19/19 +validation/linux_masked_paths.t ..................... 18/19 + not ok masked paths + error: /masktest should not be readable + +validation/mounts.t ................................... 0/1 + Skipped: 1 + TODO: mounts generation options have not been implemented + +validation/process.t ................................ 19/19 +validation/root_readonly_false.t .................... 19/19 +validation/linux_sysctl.t ........................... 19/19 +validation/linux_devices.t .......................... 19/19 +validation/linux_gid_mappings.t ..................... 18/19 + not ok gid mappings + +validation/process_oom_score_adj.t .................. 19/19 +validation/process_capabilities.t ................... 19/19 +validation/process_rlimits.t ........................ 19/19 +validation/root_readonly_true.t ...................failed to create the container +rootfsPropagation=unbindable is not supported +exit status 1 +validation/root_readonly_true.t ..................... 19/19 +validation/linux_rootfs_propagation_unbindable.t ...... 0/1 + not ok validation/linux_rootfs_propagation_unbindable.t + timeout: 30000 + file: validation/linux_rootfs_propagation_unbindable.t + command: validation/linux_rootfs_propagation_unbindable.t + args: [] + stdio: + - 0 + - pipe + - 2 + cwd: /…/go/src/github.com/opencontainers/runtime-tools + exitCode: 1 + +validation/hostname.t ...................failed to create the container +User namespace mappings specified, but USER namespace isn't enabled in the config +exit status 1 +validation/hostname.t ............................... 19/19 +validation/linux_uid_mappings.t ....................... 0/1 + not ok validation/linux_uid_mappings.t + timeout: 30000 + file: validation/linux_uid_mappings.t + command: validation/linux_uid_mappings.t + args: [] + stdio: + - 0 + - pipe + - 2 + cwd: /…/go/src/github.com/opencontainers/runtime-tools + exitCode: 1 + +total ............................................. 267/273 + + + 267 passing (31s) + 1 pending + 5 failing + +make: *** [Makefile:43: localvalidation] Error 1 +``` + +You can also run an individual test executable directly: + +```console +$ RUNTIME=runc validation/default.t +TAP version 13 +ok 1 - root filesystem +ok 2 - hostname +ok 3 - process +ok 4 - mounts +ok 5 - user +ok 6 - rlimits +ok 7 - capabilities +ok 8 - default symlinks +ok 9 - default file system +ok 10 - default devices +ok 11 - linux devices +ok 12 - linux process +ok 13 - masked paths +ok 14 - oom score adj +ok 15 - read only paths +ok 16 - rootfs propagation +ok 17 - sysctls +ok 18 - uid mappings +ok 19 - gid mappings +1..19 +``` + +If you cannot install node-tap, you can probably run the test suite with another [TAP consumer][tap-consumers]. +For example, with [`prove`][prove]: + +```console +$ sudo make TAP='prove -Q -j9' RUNTIME=runc localvalidation +RUNTIME=runc prove -Q -j9 validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t +failed to create the container +rootfsPropagation=unbindable is not supported +exit status 1 +failed to create the container +User namespace mappings specified, but USER namespace isn't enabled in the config +exit status 1 + +Test Summary Report +------------------- +validation/linux_rootfs_propagation_shared.t (Wstat: 0 Tests: 19 Failed: 1) + Failed test: 16 +validation/linux_masked_paths.t (Wstat: 0 Tests: 19 Failed: 1) + Failed test: 13 +validation/linux_rootfs_propagation_unbindable.t (Wstat: 256 Tests: 0 Failed: 0) + Non-zero exit status: 1 + Parse errors: No plan found in TAP output +validation/linux_uid_mappings.t (Wstat: 256 Tests: 0 Failed: 0) + Non-zero exit status: 1 + Parse errors: No plan found in TAP output +validation/linux_gid_mappings.t (Wstat: 0 Tests: 19 Failed: 1) + Failed test: 19 +Files=18, Tests=271, 6 wallclock secs ( 0.06 usr 0.01 sys + 0.59 cusr 0.24 csys = 0.90 CPU) +Result: FAIL +make: *** [Makefile:43: localvalidation] Error 1 +``` + + [compliance]: https://github.com/opencontainers/runtime-spec/blob/v1.0.1/spec.md +[debian-node-tap]: https://packages.debian.org/stretch/node-tap +[debian-nodejs]: https://packages.debian.org/stretch/nodejs +[gentoo-nodejs]: https://packages.gentoo.org/packages/net-libs/nodejs +[node-tap]: http://www.node-tap.org/ +[npm]: https://www.npmjs.com/ +[prove]: http://search.cpan.org/~leont/Test-Harness-3.39/bin/prove +[tap-consumers]: https://testanything.org/consumers.html From 32857b5673b0b950e55179293574386c5a5de708 Mon Sep 17 00:00:00 2001 From: Alban Crequy Date: Tue, 20 Mar 2018 12:38:48 +0100 Subject: [PATCH 3/6] docs: update validation test output Signed-off-by: Alban Crequy --- docs/runtime-compliance-testing.md | 366 ++++++++++++++++++++++------- 1 file changed, 284 insertions(+), 82 deletions(-) diff --git a/docs/runtime-compliance-testing.md b/docs/runtime-compliance-testing.md index d65dd3799..d99c8ec43 100644 --- a/docs/runtime-compliance-testing.md +++ b/docs/runtime-compliance-testing.md @@ -26,81 +26,277 @@ If we add support for alternative APIs in the future, runtime validation will ga For the command line interface, the `RUNTIME` option selects the runtime command (`funC` in the [OCI Runtime Command Line Interface](doc/command-line-interface.md)). ``` -$ sudo make RUNTIME=runc localvalidation -RUNTIME=runc tap validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t -validation/linux_rootfs_propagation_shared.t ........ 18/19 +$ sudo TAP="$(which tap)" RUNTIME=runc make localvalidation +RUNTIME=runc /home/alban/.nvm/versions/node/v9.7.1/bin/tap validation/pidfile.t validation/linux_cgroups_memory.t validation/linux_rootfs_propagation_shared.t validation/kill.t validation/linux_readonly_paths.t validation/hostname.t validation/hooks_stdin.t validation/create.t validation/poststart.t validation/linux_cgroups_network.t validation/poststop_fail.t validation/prestart_fail.t validation/linux_cgroups_relative_blkio.t validation/default.t validation/poststop.t validation/linux_seccomp.t validation/prestart.t validation/process_rlimits.t validation/linux_masked_paths.t validation/killsig.t validation/process.t validation/linux_cgroups_relative_pids.t validation/hooks.t validation/linux_rootfs_propagation_unbindable.t validation/linux_cgroups_relative_cpus.t validation/misc_props.t validation/linux_sysctl.t validation/process_oom_score_adj.t validation/linux_devices.t validation/process_capabilities_fail.t validation/start.t validation/linux_cgroups_pids.t validation/process_capabilities.t validation/poststart_fail.t validation/linux_cgroups_relative_hugetlb.t validation/mounts.t validation/linux_cgroups_hugetlb.t validation/linux_cgroups_relative_memory.t validation/state.t validation/root_readonly_true.t validation/linux_cgroups_blkio.t validation/delete.t validation/linux_cgroups_relative_network.t validation/process_rlimits_fail.t validation/linux_cgroups_cpus.t validation/linux_uid_mappings.t +validation/pidfile.t .................................. 1/1 455ms +validation/linux_cgroups_memory.t ..................... 9/9 +validation/linux_rootfs_propagation_shared.t ........ 19/20 not ok rootfs propagation error: 'rootfs should be shared, but not' +validation/kill.t ..................................... 5/5 13s +validation/linux_readonly_paths.t ................... 20/26 + Skipped: 6 + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + +validation/hostname.t ............................... 20/26 + Skipped: 6 + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + +validation/hooks_stdin.t .............................. 3/3 1s validation/create.t ................................... 4/4 -validation/default.t ................................ 19/19 -validation/linux_readonly_paths.t ................... 19/19 -validation/linux_masked_paths.t ..................... 18/19 +validation/poststart.t ................................ 0/1 + Skipped: 1 + validation/poststart.t + +validation/linux_cgroups_network.t .................... 5/5 +failed to create the container +container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:385: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: \\\"\"" +validation/poststop_fail.t ............................ 0/1 + Skipped: 1 + validation/poststop_fail.t + +validation/prestart_fail.t ............................ 0/1 + Skipped: 1 + validation/prestart_fail.t + +validation/linux_cgroups_relative_blkio.t ........... 15/15 +validation/default.t ................................ 20/26 + Skipped: 6 + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + +failed to create the container +container_linux.go:348: starting container process caused "error adding seccomp rule for syscall personality: requested action matches default action of filter" +exit status 1 +validation/poststop.t ................................. 0/1 + Skipped: 1 + validation/poststop.t + +validation/linux_seccomp.t ............................ 0/1 + not ok validation/linux_seccomp.t + error: >- + Pre-start hooks MUST be called after the `start` operation is called + + Refer to: + https://github.com/opencontainers/runtime-spec/blob/v1.0.0/config.md#prestart + +validation/prestart.t ................................. 0/1 + Skipped: 1 + validation/prestart.t + +validation/process_rlimits.t ........................ 20/26 + Skipped: 6 + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + +validation/linux_masked_paths.t ..................... 19/26 not ok masked paths error: /masktest should not be readable -validation/mounts.t ................................... 0/1 + Skipped: 6 + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + +validation/killsig.t .................................. 1/1 1s +validation/process.t ................................ 20/26 + Skipped: 6 + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + +validation/linux_cgroups_relative_pids.t .............. 3/3 +validation/hooks.t .................................... 0/1 Skipped: 1 - TODO: mounts generation options have not been implemented - -validation/process.t ................................ 19/19 -validation/root_readonly_false.t .................... 19/19 -validation/linux_sysctl.t ........................... 19/19 -validation/linux_devices.t .......................... 19/19 -validation/linux_gid_mappings.t ..................... 18/19 - not ok gid mappings - -validation/process_oom_score_adj.t .................. 19/19 -validation/process_capabilities.t ................... 19/19 -validation/process_rlimits.t ........................ 19/19 -validation/root_readonly_true.t ...................failed to create the container -rootfsPropagation=unbindable is not supported -exit status 1 -validation/root_readonly_true.t ..................... 19/19 -validation/linux_rootfs_propagation_unbindable.t ...... 0/1 - not ok validation/linux_rootfs_propagation_unbindable.t - timeout: 30000 - file: validation/linux_rootfs_propagation_unbindable.t - command: validation/linux_rootfs_propagation_unbindable.t - args: [] - stdio: - - 0 - - pipe - - 2 - cwd: /…/go/src/github.com/opencontainers/runtime-tools - exitCode: 1 + validation/hooks.t -validation/hostname.t ...................failed to create the container -User namespace mappings specified, but USER namespace isn't enabled in the config -exit status 1 -validation/hostname.t ............................... 19/19 -validation/linux_uid_mappings.t ....................... 0/1 - not ok validation/linux_uid_mappings.t +validation/linux_rootfs_propagation_unbindable.t .... 19/20 + not ok rootfs propagation + error: 'rootfs expected to be unbindable, but not' + +validation/linux_cgroups_relative_cpus.t .............. 7/7 +validation/misc_props.t ............................... 2/3 4s + not ok runtimes that are reading or processing this configuration file MUST generate an error when invalid or unsupported values are encountered + reference: >- + https://github.com/opencontainers/runtime-spec/blob/v1.0.0/config.md#valid-values + +validation/linux_sysctl.t ........................... 20/26 + Skipped: 6 + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + +validation/process_oom_score_adj.t .................. 20/26 + Skipped: 6 + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + +validation/linux_devices.t .......................... 20/26 + Skipped: 6 + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + +validation/process_capabilities_fail.t ......Any value which cannot be mapped to a relevant kernel interface MUST cause an error +Refer to: https://github.com/opencontainers/runtime-spec/blob/v1.0.0/config.md#linux-process +validation/process_capabilities_fail.t .............. 20/27 + not ok validation/process_capabilities_fail.t timeout: 30000 - file: validation/linux_uid_mappings.t - command: validation/linux_uid_mappings.t + file: validation/process_capabilities_fail.t + command: validation/process_capabilities_fail.t args: [] stdio: - 0 - pipe - 2 - cwd: /…/go/src/github.com/opencontainers/runtime-tools + cwd: /home/alban/go/src/github.com/opencontainers/runtime-tools exitCode: 1 -total ............................................. 267/273 + Skipped: 6 + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + +validation/start.t .....exit status 2 +validation/start.t .................................... 6/7 + not ok test count !== plan + +++ found + --- wanted + -1 + +6 + results: + ok: false + count: 6 + pass: 6 + fail: 1 + bailout: false + todo: 0 + skip: 0 + plan: + start: null + end: null + skipAll: false + skipReason: '' + comment: '' + failures: + - tapError: no plan + +validation/linux_cgroups_pids.t ....................... 3/3 +validation/process_capabilities.t ....................failed to create the container +container_linux.go:376: running poststart hook 0 caused "error running hook: exit status 1, stdout: , stderr: " +validation/process_capabilities.t ................... 20/20 +validation/poststart_fail.t ........................... 0/1 + Skipped: 1 + validation/poststart_fail.t +validation/linux_cgroups_relative_hugetlb.t ........... 4/4 +validation/mounts.t ................................. 20/26 + Skipped: 6 + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW - 267 passing (31s) - 1 pending - 5 failing +validation/linux_cgroups_hugetlb.t .................... 0/1 + Skipped: 1 + validation/linux_cgroups_hugetlb.t no tests found + +validation/linux_cgroups_relative_memory.t ............ 9/9 +validation/state.t .................................... 3/3 +validation/root_readonly_true.t ..................... 20/26 + Skipped: 6 + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + +validation/linux_cgroups_blkio.t .................... 15/15 +validation/delete.t ................................... 3/5 22s + not ok attempting to `delete` a container that is not `stopped` MUST generate an error + reference: 'https://github.com/opencontainers/runtime-spec/blob/v1.0.0/runtime.md#delete' + + not ok attempting to `delete` a container that is not `stopped` MUST have no effect on the container + error: exit status 1 + reference: 'https://github.com/opencontainers/runtime-spec/blob/v1.0.0/runtime.md#delete' + stderr: | + container "dd7f1685-1ea8-468c-8e6a-37f5f9f9ca64" does not exist + +validation/linux_cgroups_relative_network.t .....failed to create the container +wrong rlimit value: RLIMIT_TEST +validation/linux_cgroups_relative_network.t ........... 5/5 +validation/process_rlimits_fail.t ..................... 0/1 + Skipped: 1 + validation/process_rlimits_fail.t no tests found + +validation/linux_cgroups_cpus.t ....................... 0/1 + Skipped: 1 + validation/linux_cgroups_cpus.t no tests found -make: *** [Makefile:43: localvalidation] Error 1 +validation/linux_uid_mappings.t ..................... 20/26 + Skipped: 6 + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + +total ............................................. 420/517 + + + 420 passing (1m) + 88 pending + 9 failing + +make: *** [Makefile:44: localvalidation] Error 1 ``` You can also run an individual test executable directly: ```console -$ RUNTIME=runc validation/default.t +$ sudo RUNTIME=runc validation/default.t TAP version 13 ok 1 - root filesystem ok 2 - hostname @@ -116,44 +312,50 @@ ok 11 - linux devices ok 12 - linux process ok 13 - masked paths ok 14 - oom score adj -ok 15 - read only paths -ok 16 - rootfs propagation -ok 17 - sysctls -ok 18 - uid mappings -ok 19 - gid mappings -1..19 +ok 15 # SKIP syscall action SCMP_ACT_ALLOW +ok 16 # SKIP syscall action SCMP_ACT_ALLOW +ok 17 # SKIP syscall action SCMP_ACT_ALLOW +ok 18 # SKIP syscall action SCMP_ACT_ALLOW +ok 19 # SKIP syscall action SCMP_ACT_ALLOW +ok 20 # SKIP syscall action SCMP_ACT_ALLOW +ok 21 - seccomp +ok 22 - read only paths +ok 23 - rootfs propagation +ok 24 - sysctls +ok 25 - uid mappings +ok 26 - gid mappings +1..26 +``` + +or with the environment variable `VALIDATION_TESTS`: + +```console +$ sudo make TAP=$(which tap) RUNTIME=runc VALIDATION_TESTS=validation/default.t localvalidation +RUNTIME=runc /home/alban/.nvm/versions/node/v9.7.1/bin/tap validation/default.t +validation/default.t ................................ 20/26 + Skipped: 6 + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + syscall action SCMP_ACT_ALLOW + +total ............................................... 20/26 + + 20 passing (257.078ms) + 6 pending ``` If you cannot install node-tap, you can probably run the test suite with another [TAP consumer][tap-consumers]. For example, with [`prove`][prove]: ```console -$ sudo make TAP='prove -Q -j9' RUNTIME=runc localvalidation -RUNTIME=runc prove -Q -j9 validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t -failed to create the container -rootfsPropagation=unbindable is not supported -exit status 1 -failed to create the container -User namespace mappings specified, but USER namespace isn't enabled in the config -exit status 1 - -Test Summary Report -------------------- -validation/linux_rootfs_propagation_shared.t (Wstat: 0 Tests: 19 Failed: 1) - Failed test: 16 -validation/linux_masked_paths.t (Wstat: 0 Tests: 19 Failed: 1) - Failed test: 13 -validation/linux_rootfs_propagation_unbindable.t (Wstat: 256 Tests: 0 Failed: 0) - Non-zero exit status: 1 - Parse errors: No plan found in TAP output -validation/linux_uid_mappings.t (Wstat: 256 Tests: 0 Failed: 0) - Non-zero exit status: 1 - Parse errors: No plan found in TAP output -validation/linux_gid_mappings.t (Wstat: 0 Tests: 19 Failed: 1) - Failed test: 19 -Files=18, Tests=271, 6 wallclock secs ( 0.06 usr 0.01 sys + 0.59 cusr 0.24 csys = 0.90 CPU) -Result: FAIL -make: *** [Makefile:43: localvalidation] Error 1 +$ sudo make TAP="prove -Q -j9" RUNTIME=runc VALIDATION_TESTS="validation/default.t validation/linux_cgroups_memory.t" localvalidation +RUNTIME=runc prove -Q -j9 validation/default.t validation/linux_cgroups_memory.t +All tests successful. +Files=2, Tests=35, 1 wallclock secs ( 0.03 usr 0.00 sys + 0.12 cusr 0.12 csys = 0.27 CPU) +Result: PASS ``` From 25522e109132c57b427b45a9884b3bc110cff975 Mon Sep 17 00:00:00 2001 From: Alban Crequy Date: Tue, 20 Mar 2018 14:46:15 +0100 Subject: [PATCH 4/6] docs: how to do a release Signed-off-by: Alban Crequy --- docs/devel/release.md | 64 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 docs/devel/release.md diff --git a/docs/devel/release.md b/docs/devel/release.md new file mode 100644 index 000000000..9a3fa8647 --- /dev/null +++ b/docs/devel/release.md @@ -0,0 +1,64 @@ +# runtime-tools release guide + +## Release process + +This section shows how to perform a release of runtime-tools. +The following example assumes we're going from version 0.5.0 (`v0.5.0`) to 0.6.0 (`v0.6.0`). + +Let's get started: + +- Start at the relevant milestone on GitHub (e.g. https://github.com/opencontainers/runtime-tools/milestones/v0.6.0): ensure all referenced issues are closed (or moved elsewhere, if they're not done). Close the milestone. +- runtime-tools does not use a [roadmap file](https://github.com/opencontainers/runtime-tools/issues/465) but GitHub milestones. Update the [other milestones](https://github.com/opencontainers/runtime-tools/milestones), if necessary +- Branch from the latest master, make sure your git status is clean +- Update the [VERSION](https://github.com/opencontainers/runtime-tools/blob/master/VERSION) +- Update the [release notes][changelog]. + Try to capture most of the salient changes since the last release, but don't go into unnecessary detail (better to link/reference the documentation wherever possible). + +Ensure the branch is correct: + +- Ensure the build is clean! + - `git clean -ffdx && make && make test` should work +- Integration tests on CI should be green +- Check the version of the binaries: + - Check `./oci-runtime-tool --version` + - Check `./runtimetest --version` + +Once everything is fine: + +- File a PR +- Ensure the CI on the release PR is green +- Send an email to the [mailing list][mailinglist] ([example for v0.5.0](https://groups.google.com/a/opencontainers.org/forum/#!topic/dev/iuWpWUai4_I)) and get reviews from other [maintainers][maintainers]. +- Once the maintainers agree, merge the PR + +Sign a tagged release and push it to GitHub: + +- Add a signed tag: `git tag -s v0.6.0 -m "release v0.6.0"` +- Push the tag to GitHub: `git push origin v0.6.0` + +Now we switch to the GitHub web UI to conduct the release: + +- Start a [new release][gh-new-release] on Github +- Tag "v0.6.0", release title "v0.6.0" +- Copy-paste the release notes you added earlier in [CHANGELOG.md][changelog] +- Attach the release. + This is a simple tarball: + +``` +export VER="1.2.0" +export NAME="runtime-tools-v$VER" +mkdir -p $NAME/validation +cp oci-runtime-tool $NAME/ +cp validation/*.t $NAME/validation/ +sudo chown -R root:root $NAME/ +tar czvf $NAME.tar.gz --numeric-owner $NAME/ +``` + +- Publish the release! + +- Clean your git tree: `sudo git clean -ffdx`. + +[changelog]: https://github.com/opencontainers/runtime-tools/blob/master/CHANGELOG.md +[maintainers]: https://github.com/opencontainers/runtime-tools/blob/master/MAINTAINERS +[mailinglist]: https://groups.google.com/a/opencontainers.org/forum/#!forum/dev +[gh-new-release]: https://github.com/opencontainers/runtime-tools/releases/new + From ea0a3b26db70a23e1a14481128ae9248b11ac7bf Mon Sep 17 00:00:00 2001 From: Alban Crequy Date: Tue, 20 Mar 2018 14:57:11 +0100 Subject: [PATCH 5/6] docs: how to run tests from a release Signed-off-by: Alban Crequy --- docs/runtime-compliance-testing.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/runtime-compliance-testing.md b/docs/runtime-compliance-testing.md index d99c8ec43..1d1c25834 100644 --- a/docs/runtime-compliance-testing.md +++ b/docs/runtime-compliance-testing.md @@ -6,7 +6,7 @@ In order to be tested for [compliance][], runtimes MUST support at least one of * Version 1.0.1 of the [OCI Runtime Command Line Interface](command-line-interface.md). -## Running the runtime validation suite from source +## Running the runtime validation suite The runtime validation suite uses [node-tap][], which is packaged for some distributions (for example, it is in [Debian's `node-tap` package][debian-node-tap]). If your distribution does not package node-tap, you can install [npm][] (for example, from [Gentoo's `nodejs` package][gentoo-nodejs]) and use it: @@ -14,6 +14,16 @@ If your distribution does not package node-tap, you can install [npm][] (for exa ```console $ npm install tap ``` +### From a release + +Check if your release has pre-compiled tests on the [release page][releases] page. + +``` +$ tar xf runtime-tools-v0.6.0.tar.gz +$ RUNTIME=runc tap ./runtime-tools-v0.6.0/validation/*.t +``` + +### From source Build the validation executables: @@ -367,3 +377,4 @@ Result: PASS [npm]: https://www.npmjs.com/ [prove]: http://search.cpan.org/~leont/Test-Harness-3.39/bin/prove [tap-consumers]: https://testanything.org/consumers.html +[releases]: https://github.com/opencontainers/runtime-tools/releases From 418cabb0091e8c6707ded04169b327150ff9d1ec Mon Sep 17 00:00:00 2001 From: Dongsu Park Date: Fri, 24 Aug 2018 11:44:42 +0200 Subject: [PATCH 6/6] docs: Address review comments Address review comments written by @wking. * Add a blank line between non-manpage refs and manpage refs. * Add backticks for `git status`. * Append a punctuation mark to each list item. * Use `pull request` instead of `PR`. * Specify `console` format to the example commands * Use relative links instead of static URIs * Remove a trailing blank * Remove trailing whitespaces * Run with sudo whenever it's possible * Use correct relative links * Move sections for running runtime validation one level up to the section `Running the runtime validation suite`. * Use `command -v` instead of `which`. * Elaborate why we should specify the TAP variable. Signed-off-by: Dongsu Park --- README.md | 3 +- docs/devel/release.md | 51 +++++++++++++-------------- docs/runtime-compliance-testing.md | 56 ++++++++++++++++-------------- 3 files changed, 56 insertions(+), 54 deletions(-) diff --git a/README.md b/README.md index ffbbdd0e2..da903b7a0 100644 --- a/README.md +++ b/README.md @@ -30,11 +30,12 @@ INFO[0000] Bundle validation succeeded. ## Testing OCI runtimes -Use the [runtime validation suite](doc/runtime-compliance-testing.md). +Use the [runtime validation suite](docs/runtime-compliance-testing.md). [bundle]: https://github.com/opencontainers/runtime-spec/blob/master/bundle.md [config.json]: https://github.com/opencontainers/runtime-spec/blob/master/config.md [runC]: https://github.com/opencontainers/runc [runtime-spec]: https://github.com/opencontainers/runtime-spec + [generate.1]: man/oci-runtime-tool-generate.1.md [validate.1]: man/oci-runtime-tool-validate.1.md diff --git a/docs/devel/release.md b/docs/devel/release.md index 9a3fa8647..81785ee0c 100644 --- a/docs/devel/release.md +++ b/docs/devel/release.md @@ -8,57 +8,56 @@ The following example assumes we're going from version 0.5.0 (`v0.5.0`) to 0.6.0 Let's get started: - Start at the relevant milestone on GitHub (e.g. https://github.com/opencontainers/runtime-tools/milestones/v0.6.0): ensure all referenced issues are closed (or moved elsewhere, if they're not done). Close the milestone. -- runtime-tools does not use a [roadmap file](https://github.com/opencontainers/runtime-tools/issues/465) but GitHub milestones. Update the [other milestones](https://github.com/opencontainers/runtime-tools/milestones), if necessary -- Branch from the latest master, make sure your git status is clean -- Update the [VERSION](https://github.com/opencontainers/runtime-tools/blob/master/VERSION) +- runtime-tools does not use a [roadmap file](https://github.com/opencontainers/runtime-tools/issues/465) but GitHub milestones. Update the [other milestones](https://github.com/opencontainers/runtime-tools/milestones), if necessary. +- Branch from the latest master, make sure your `git status` is clean. +- Update the [VERSION](https://github.com/opencontainers/runtime-tools/blob/master/VERSION). - Update the [release notes][changelog]. Try to capture most of the salient changes since the last release, but don't go into unnecessary detail (better to link/reference the documentation wherever possible). Ensure the branch is correct: - Ensure the build is clean! - - `git clean -ffdx && make && make test` should work -- Integration tests on CI should be green + - `git clean -ffdx && make && make test` should work. +- Integration tests on CI should be green. - Check the version of the binaries: - - Check `./oci-runtime-tool --version` - - Check `./runtimetest --version` + - Check `./oci-runtime-tool --version`. + - Check `./runtimetest --version`. Once everything is fine: -- File a PR -- Ensure the CI on the release PR is green +- File a pull request. +- Ensure the CI on the release PR is green. - Send an email to the [mailing list][mailinglist] ([example for v0.5.0](https://groups.google.com/a/opencontainers.org/forum/#!topic/dev/iuWpWUai4_I)) and get reviews from other [maintainers][maintainers]. -- Once the maintainers agree, merge the PR +- Once the maintainers agree, merge the PR. Sign a tagged release and push it to GitHub: -- Add a signed tag: `git tag -s v0.6.0 -m "release v0.6.0"` -- Push the tag to GitHub: `git push origin v0.6.0` +- Add a signed tag: `git tag -s v0.6.0 -m "release v0.6.0"`. +- Push the tag to GitHub: `git push origin v0.6.0`. Now we switch to the GitHub web UI to conduct the release: -- Start a [new release][gh-new-release] on Github -- Tag "v0.6.0", release title "v0.6.0" -- Copy-paste the release notes you added earlier in [CHANGELOG.md][changelog] +- Start a [new release][gh-new-release] on Github. +- Tag "v0.6.0", release title "v0.6.0". +- Copy-paste the release notes you added earlier in [CHANGELOG.md][changelog]. - Attach the release. This is a simple tarball: -``` -export VER="1.2.0" -export NAME="runtime-tools-v$VER" -mkdir -p $NAME/validation -cp oci-runtime-tool $NAME/ -cp validation/*.t $NAME/validation/ -sudo chown -R root:root $NAME/ -tar czvf $NAME.tar.gz --numeric-owner $NAME/ +```console +$ export VER="1.2.0" +$ export NAME="runtime-tools-v$VER" +$ mkdir -p $NAME/validation +$ cp oci-runtime-tool $NAME/ +$ cp validation/*.t $NAME/validation/ +$ sudo chown -R root:root $NAME/ +$ tar czvf $NAME.tar.gz --numeric-owner $NAME/ ``` - Publish the release! - Clean your git tree: `sudo git clean -ffdx`. -[changelog]: https://github.com/opencontainers/runtime-tools/blob/master/CHANGELOG.md -[maintainers]: https://github.com/opencontainers/runtime-tools/blob/master/MAINTAINERS +[changelog]: ../../CHANGELOG.md +[maintainers]: ../../MAINTAINERS [mailinglist]: https://groups.google.com/a/opencontainers.org/forum/#!forum/dev [gh-new-release]: https://github.com/opencontainers/runtime-tools/releases/new - diff --git a/docs/runtime-compliance-testing.md b/docs/runtime-compliance-testing.md index 1d1c25834..fec73764e 100644 --- a/docs/runtime-compliance-testing.md +++ b/docs/runtime-compliance-testing.md @@ -14,29 +14,15 @@ If your distribution does not package node-tap, you can install [npm][] (for exa ```console $ npm install tap ``` -### From a release - -Check if your release has pre-compiled tests on the [release page][releases] page. - -``` -$ tar xf runtime-tools-v0.6.0.tar.gz -$ RUNTIME=runc tap ./runtime-tools-v0.6.0/validation/*.t -``` - -### From source -Build the validation executables: - -```console -$ make runtimetest validation-executables -``` - -Runtime validation currently [only supports](docs/runtime-compliance-testing.md) the [OCI Runtime Command Line Interface](doc/command-line-interface.md). +Runtime validation currently [only supports](runtime-compliance-testing.md) the [OCI Runtime Command Line Interface](command-line-interface.md). If we add support for alternative APIs in the future, runtime validation will gain an option to select the desired runtime API. -For the command line interface, the `RUNTIME` option selects the runtime command (`funC` in the [OCI Runtime Command Line Interface](doc/command-line-interface.md)). +For the command line interface, the `RUNTIME` option selects the runtime command (`funC` in the [OCI Runtime Command Line Interface](command-line-interface.md)). + +Note that we should specify the `TAP` variable to the normal user's tap command, because usually the normal user had a newer version of `tap` installed locally via npm, for example, under `$HOME/node_modules/.bin`. ``` -$ sudo TAP="$(which tap)" RUNTIME=runc make localvalidation +$ sudo TAP="$(command -v tap)" RUNTIME=runc make localvalidation RUNTIME=runc /home/alban/.nvm/versions/node/v9.7.1/bin/tap validation/pidfile.t validation/linux_cgroups_memory.t validation/linux_rootfs_propagation_shared.t validation/kill.t validation/linux_readonly_paths.t validation/hostname.t validation/hooks_stdin.t validation/create.t validation/poststart.t validation/linux_cgroups_network.t validation/poststop_fail.t validation/prestart_fail.t validation/linux_cgroups_relative_blkio.t validation/default.t validation/poststop.t validation/linux_seccomp.t validation/prestart.t validation/process_rlimits.t validation/linux_masked_paths.t validation/killsig.t validation/process.t validation/linux_cgroups_relative_pids.t validation/hooks.t validation/linux_rootfs_propagation_unbindable.t validation/linux_cgroups_relative_cpus.t validation/misc_props.t validation/linux_sysctl.t validation/process_oom_score_adj.t validation/linux_devices.t validation/process_capabilities_fail.t validation/start.t validation/linux_cgroups_pids.t validation/process_capabilities.t validation/poststart_fail.t validation/linux_cgroups_relative_hugetlb.t validation/mounts.t validation/linux_cgroups_hugetlb.t validation/linux_cgroups_relative_memory.t validation/state.t validation/root_readonly_true.t validation/linux_cgroups_blkio.t validation/delete.t validation/linux_cgroups_relative_network.t validation/process_rlimits_fail.t validation/linux_cgroups_cpus.t validation/linux_uid_mappings.t validation/pidfile.t .................................. 1/1 455ms validation/linux_cgroups_memory.t ..................... 9/9 @@ -101,7 +87,7 @@ validation/linux_seccomp.t ............................ 0/1 not ok validation/linux_seccomp.t error: >- Pre-start hooks MUST be called after the `start` operation is called - + Refer to: https://github.com/opencontainers/runtime-spec/blob/v1.0.0/config.md#prestart @@ -208,10 +194,10 @@ validation/process_capabilities_fail.t .............. 20/27 validation/start.t .....exit status 2 validation/start.t .................................... 6/7 not ok test count !== plan - +++ found - --- wanted - -1 - +6 + +++ found + --- wanted + -1 + +6 results: ok: false count: 6 @@ -266,7 +252,7 @@ validation/linux_cgroups_blkio.t .................... 15/15 validation/delete.t ................................... 3/5 22s not ok attempting to `delete` a container that is not `stopped` MUST generate an error reference: 'https://github.com/opencontainers/runtime-spec/blob/v1.0.0/runtime.md#delete' - + not ok attempting to `delete` a container that is not `stopped` MUST have no effect on the container error: exit status 1 reference: 'https://github.com/opencontainers/runtime-spec/blob/v1.0.0/runtime.md#delete' @@ -294,7 +280,7 @@ validation/linux_uid_mappings.t ..................... 20/26 syscall action SCMP_ACT_ALLOW total ............................................. 420/517 - + 420 passing (1m) 88 pending @@ -340,7 +326,7 @@ ok 26 - gid mappings or with the environment variable `VALIDATION_TESTS`: ```console -$ sudo make TAP=$(which tap) RUNTIME=runc VALIDATION_TESTS=validation/default.t localvalidation +$ sudo make TAP=$(command -v tap) RUNTIME=runc VALIDATION_TESTS=validation/default.t localvalidation RUNTIME=runc /home/alban/.nvm/versions/node/v9.7.1/bin/tap validation/default.t validation/default.t ................................ 20/26 Skipped: 6 @@ -368,6 +354,22 @@ Files=2, Tests=35, 1 wallclock secs ( 0.03 usr 0.00 sys + 0.12 cusr 0.12 csy Result: PASS ``` +### From a release + +Check if your release has pre-compiled tests on the [release page][releases] page. + +```console +$ tar xf runtime-tools-v0.6.0.tar.gz +$ sudo RUNTIME=runc tap ./runtime-tools-v0.6.0/validation/*.t +``` + +### From source + +Build the validation executables: + +```console +$ make runtimetest validation-executables +``` [compliance]: https://github.com/opencontainers/runtime-spec/blob/v1.0.1/spec.md [debian-node-tap]: https://packages.debian.org/stretch/node-tap