-
Notifications
You must be signed in to change notification settings - Fork 144
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Running validation tests from a release #610
base: master
Are you sure you want to change the base?
Changes from 1 commit
7aaee9c
294ca2a
32857b5
25522e1
ea0a3b2
418cabb
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
Signed-off-by: Alban Crequy <alban@kinvolk.io>
- Loading branch information
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -30,99 +30,11 @@ INFO[0000] Bundle validation succeeded. | |
|
||
## Testing OCI runtimes | ||
|
||
The runtime validation suite uses [node-tap][], which is packaged for some distributions (for example, it is in [Debian's `node-tap` package][debian-node-tap]). | ||
If your distribution does not package node-tap, you can install [npm][] (for example, from [Gentoo's `nodejs` package][gentoo-nodejs]) and use it: | ||
|
||
```console | ||
$ npm install tap | ||
``` | ||
|
||
Build the validation executables: | ||
|
||
```console | ||
$ make runtimetest validation-executables | ||
``` | ||
|
||
Runtime validation currently [only supports](docs/runtime-compliance-testing.md) the [OCI Runtime Command Line Interface](docs/command-line-interface.md). | ||
If we add support for alternative APIs in the future, runtime validation will gain an option to select the desired runtime API. | ||
For the command line interface, the `RUNTIME` option selects the runtime command (`funC` in the [OCI Runtime Command Line Interface](docs/command-line-interface.md)). | ||
|
||
``` | ||
$ sudo make RUNTIME=runc localvalidation | ||
RUNTIME=runc tap validation/pidfile.t validation/linux_cgroups_hugetlb.t validation/linux_cgroups_memory.t validation/linux_rootfs_propagation_shared.t validation/kill.t validation/create.t validation/poststart.t validation/linux_cgroups_network.t validation/poststop_fail.t validation/linux_readonly_paths.t validation/prestart_fail.t validation/hooks_stdin.t validation/default.t validation/linux_masked_paths.t validation/poststop.t validation/misc_props.t validation/prestart.t validation/poststart_fail.t validation/mounts.t validation/linux_cgroups_relative_pids.t validation/process_user.t validation/process.t validation/hooks.t validation/process_capabilities_fail.t validation/process_rlimits_fail.t validation/linux_cgroups_relative_cpus.t validation/process_rlimits.t validation/linux_cgroups_relative_blkio.t validation/linux_sysctl.t validation/linux_seccomp.t validation/linux_devices.t validation/start.t validation/linux_cgroups_pids.t validation/process_capabilities.t validation/process_oom_score_adj.t validation/linux_cgroups_relative_hugetlb.t validation/linux_cgroups_cpus.t validation/linux_cgroups_relative_memory.t validation/state.t validation/root_readonly_true.t validation/linux_cgroups_blkio.t validation/linux_rootfs_propagation_unbindable.t validation/delete.t validation/linux_cgroups_relative_network.t validation/hostname.t validation/killsig.t validation/linux_uid_mappings.t | ||
validation/pidfile.t .failed to create the container | ||
container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 56892210544640 to hugetlb.1GB.limit_in_bytes: open /sys/fs/cgroup/hugetlb/cgrouptest/hugetlb.1GB.limit_in_bytes: permission denied\\\"\"" | ||
exit status 1 | ||
validation/pidfile.t .................................. 1/1 315ms | ||
validation/linux_cgroups_hugetlb.t .................... 0/1 | ||
not ok validation/linux_cgroups_hugetlb.t | ||
timeout: 30000 | ||
file: validation/linux_cgroups_hugetlb.t | ||
command: validation/linux_cgroups_hugetlb.t | ||
args: [] | ||
stdio: | ||
- 0 | ||
- pipe | ||
- 2 | ||
cwd: /…/go/src/github.com/opencontainers/runtime-tools | ||
exitCode: 1 | ||
|
||
validation/linux_cgroups_memory.t ..................... 9/9 | ||
validation/linux_rootfs_propagation_shared.t ...... 252/282 | ||
not ok shared root propagation exposes "/target348456609/mount892511628/example376408222" | ||
|
||
Skipped: 29 | ||
/dev/null (default device) has unconfigured permissions | ||
… | ||
total ........................................... 4381/4962 | ||
|
||
|
||
4381 passing (1m) | ||
567 pending | ||
14 failing | ||
|
||
make: *** [Makefile:44: localvalidation] Error 1 | ||
``` | ||
|
||
You can also run an individual test executable directly: | ||
|
||
```console | ||
$ RUNTIME=runc validation/default.t | ||
TAP version 13 | ||
ok 1 - has expected hostname | ||
--- | ||
{ | ||
"actual": "mrsdalloway", | ||
"expected": "mrsdalloway" | ||
} | ||
... | ||
… | ||
ok 287 # SKIP linux.gidMappings not set | ||
1..287 | ||
``` | ||
|
||
If you cannot install node-tap, you can probably run the test suite with another [TAP consumer][tap-consumers]. | ||
For example, with [`prove`][prove]: | ||
|
||
```console | ||
$ sudo make TAP='prove -Q -j9' RUNTIME=runc VALIDATION_TESTS=validation/pidfile.t localvalidation | ||
RUNTIME=runc prove -Q -j9 validation/pidfile.t | ||
All tests successful. | ||
Files=1, Tests=1, 0 wallclock secs ( 0.01 usr 0.01 sys + 0.03 cusr 0.03 csys = 0.08 CPU) | ||
Result: PASS | ||
``` | ||
Use the [runtime validation suite](doc/runtime-compliance-testing.md). | ||
|
||
[bundle]: https://github.com/opencontainers/runtime-spec/blob/master/bundle.md | ||
[config.json]: https://github.com/opencontainers/runtime-spec/blob/master/config.md | ||
[debian-node-tap]: https://packages.debian.org/stretch/node-tap | ||
[debian-nodejs]: https://packages.debian.org/stretch/nodejs | ||
[gentoo-nodejs]: https://packages.gentoo.org/packages/net-libs/nodejs | ||
[node-tap]: http://www.node-tap.org/ | ||
[npm]: https://www.npmjs.com/ | ||
[prove]: http://search.cpan.org/~leont/Test-Harness-3.39/bin/prove | ||
[runC]: https://github.com/opencontainers/runc | ||
[runtime-spec]: https://github.com/opencontainers/runtime-spec | ||
[tap-consumers]: https://testanything.org/consumers.html | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't know if we have a runtime-tools policy on this point, but runtime-spec prefers a blank line between the non-man-page and man-page references. |
||
[generate.1]: man/oci-runtime-tool-generate.1.md | ||
[validate.1]: man/oci-runtime-tool-validate.1.md |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,4 +6,162 @@ In order to be tested for [compliance][], runtimes MUST support at least one of | |
|
||
* Version 1.0.1 of the [OCI Runtime Command Line Interface](command-line-interface.md). | ||
|
||
## Running the runtime validation suite from source | ||
|
||
The runtime validation suite uses [node-tap][], which is packaged for some distributions (for example, it is in [Debian's `node-tap` package][debian-node-tap]). | ||
If your distribution does not package node-tap, you can install [npm][] (for example, from [Gentoo's `nodejs` package][gentoo-nodejs]) and use it: | ||
|
||
```console | ||
$ npm install tap | ||
``` | ||
|
||
Build the validation executables: | ||
|
||
```console | ||
$ make runtimetest validation-executables | ||
``` | ||
|
||
Runtime validation currently [only supports](docs/runtime-compliance-testing.md) the [OCI Runtime Command Line Interface](doc/command-line-interface.md). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These relative links (and some later ones in this file) need to be updated now that you've moved the text under Also, this information is generic for runtime-validation; it's not specific to the “From source” approach. Can you shift it (and some of the text from the next paragraph) up into the “Running the runtime validation suite” section? |
||
If we add support for alternative APIs in the future, runtime validation will gain an option to select the desired runtime API. | ||
For the command line interface, the `RUNTIME` option selects the runtime command (`funC` in the [OCI Runtime Command Line Interface](doc/command-line-interface.md)). | ||
|
||
``` | ||
$ sudo make RUNTIME=runc localvalidation | ||
RUNTIME=runc tap validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t | ||
validation/linux_rootfs_propagation_shared.t ........ 18/19 | ||
not ok rootfs propagation | ||
error: 'rootfs should be shared, but not' | ||
|
||
validation/create.t ................................... 4/4 | ||
validation/default.t ................................ 19/19 | ||
validation/linux_readonly_paths.t ................... 19/19 | ||
validation/linux_masked_paths.t ..................... 18/19 | ||
not ok masked paths | ||
error: /masktest should not be readable | ||
|
||
validation/mounts.t ................................... 0/1 | ||
Skipped: 1 | ||
TODO: mounts generation options have not been implemented | ||
|
||
validation/process.t ................................ 19/19 | ||
validation/root_readonly_false.t .................... 19/19 | ||
validation/linux_sysctl.t ........................... 19/19 | ||
validation/linux_devices.t .......................... 19/19 | ||
validation/linux_gid_mappings.t ..................... 18/19 | ||
not ok gid mappings | ||
|
||
validation/process_oom_score_adj.t .................. 19/19 | ||
validation/process_capabilities.t ................... 19/19 | ||
validation/process_rlimits.t ........................ 19/19 | ||
validation/root_readonly_true.t ...................failed to create the container | ||
rootfsPropagation=unbindable is not supported | ||
exit status 1 | ||
validation/root_readonly_true.t ..................... 19/19 | ||
validation/linux_rootfs_propagation_unbindable.t ...... 0/1 | ||
not ok validation/linux_rootfs_propagation_unbindable.t | ||
timeout: 30000 | ||
file: validation/linux_rootfs_propagation_unbindable.t | ||
command: validation/linux_rootfs_propagation_unbindable.t | ||
args: [] | ||
stdio: | ||
- 0 | ||
- pipe | ||
- 2 | ||
cwd: /…/go/src/github.com/opencontainers/runtime-tools | ||
exitCode: 1 | ||
|
||
validation/hostname.t ...................failed to create the container | ||
User namespace mappings specified, but USER namespace isn't enabled in the config | ||
exit status 1 | ||
validation/hostname.t ............................... 19/19 | ||
validation/linux_uid_mappings.t ....................... 0/1 | ||
not ok validation/linux_uid_mappings.t | ||
timeout: 30000 | ||
file: validation/linux_uid_mappings.t | ||
command: validation/linux_uid_mappings.t | ||
args: [] | ||
stdio: | ||
- 0 | ||
- pipe | ||
- 2 | ||
cwd: /…/go/src/github.com/opencontainers/runtime-tools | ||
exitCode: 1 | ||
|
||
total ............................................. 267/273 | ||
|
||
|
||
267 passing (31s) | ||
1 pending | ||
5 failing | ||
|
||
make: *** [Makefile:43: localvalidation] Error 1 | ||
``` | ||
|
||
You can also run an individual test executable directly: | ||
|
||
```console | ||
$ RUNTIME=runc validation/default.t | ||
TAP version 13 | ||
ok 1 - root filesystem | ||
ok 2 - hostname | ||
ok 3 - process | ||
ok 4 - mounts | ||
ok 5 - user | ||
ok 6 - rlimits | ||
ok 7 - capabilities | ||
ok 8 - default symlinks | ||
ok 9 - default file system | ||
ok 10 - default devices | ||
ok 11 - linux devices | ||
ok 12 - linux process | ||
ok 13 - masked paths | ||
ok 14 - oom score adj | ||
ok 15 - read only paths | ||
ok 16 - rootfs propagation | ||
ok 17 - sysctls | ||
ok 18 - uid mappings | ||
ok 19 - gid mappings | ||
1..19 | ||
``` | ||
|
||
If you cannot install node-tap, you can probably run the test suite with another [TAP consumer][tap-consumers]. | ||
For example, with [`prove`][prove]: | ||
|
||
```console | ||
$ sudo make TAP='prove -Q -j9' RUNTIME=runc localvalidation | ||
RUNTIME=runc prove -Q -j9 validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t | ||
failed to create the container | ||
rootfsPropagation=unbindable is not supported | ||
exit status 1 | ||
failed to create the container | ||
User namespace mappings specified, but USER namespace isn't enabled in the config | ||
exit status 1 | ||
|
||
Test Summary Report | ||
------------------- | ||
validation/linux_rootfs_propagation_shared.t (Wstat: 0 Tests: 19 Failed: 1) | ||
Failed test: 16 | ||
validation/linux_masked_paths.t (Wstat: 0 Tests: 19 Failed: 1) | ||
Failed test: 13 | ||
validation/linux_rootfs_propagation_unbindable.t (Wstat: 256 Tests: 0 Failed: 0) | ||
Non-zero exit status: 1 | ||
Parse errors: No plan found in TAP output | ||
validation/linux_uid_mappings.t (Wstat: 256 Tests: 0 Failed: 0) | ||
Non-zero exit status: 1 | ||
Parse errors: No plan found in TAP output | ||
validation/linux_gid_mappings.t (Wstat: 0 Tests: 19 Failed: 1) | ||
Failed test: 19 | ||
Files=18, Tests=271, 6 wallclock secs ( 0.06 usr 0.01 sys + 0.59 cusr 0.24 csys = 0.90 CPU) | ||
Result: FAIL | ||
make: *** [Makefile:43: localvalidation] Error 1 | ||
``` | ||
|
||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You only need one blank line here. |
||
[compliance]: https://github.com/opencontainers/runtime-spec/blob/v1.0.1/spec.md | ||
[debian-node-tap]: https://packages.debian.org/stretch/node-tap | ||
[debian-nodejs]: https://packages.debian.org/stretch/nodejs | ||
[gentoo-nodejs]: https://packages.gentoo.org/packages/net-libs/nodejs | ||
[node-tap]: http://www.node-tap.org/ | ||
[npm]: https://www.npmjs.com/ | ||
[prove]: http://search.cpan.org/~leont/Test-Harness-3.39/bin/prove | ||
[tap-consumers]: https://testanything.org/consumers.html |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
doc -> docs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@q384566678 Good catch. Fixed. Thanks!