Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running validation tests from a release #610

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
docs: move compliance testing to separate doc
Signed-off-by: Alban Crequy <alban@kinvolk.io>
  • Loading branch information
alban authored and Dongsu Park committed Aug 22, 2018
commit 294ca2a32459d34228156fccb8c91f42822d2b95
90 changes: 1 addition & 89 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,99 +30,11 @@ INFO[0000] Bundle validation succeeded.

## Testing OCI runtimes

The runtime validation suite uses [node-tap][], which is packaged for some distributions (for example, it is in [Debian's `node-tap` package][debian-node-tap]).
If your distribution does not package node-tap, you can install [npm][] (for example, from [Gentoo's `nodejs` package][gentoo-nodejs]) and use it:

```console
$ npm install tap
```

Build the validation executables:

```console
$ make runtimetest validation-executables
```

Runtime validation currently [only supports](docs/runtime-compliance-testing.md) the [OCI Runtime Command Line Interface](docs/command-line-interface.md).
If we add support for alternative APIs in the future, runtime validation will gain an option to select the desired runtime API.
For the command line interface, the `RUNTIME` option selects the runtime command (`funC` in the [OCI Runtime Command Line Interface](docs/command-line-interface.md)).

```
$ sudo make RUNTIME=runc localvalidation
RUNTIME=runc tap validation/pidfile.t validation/linux_cgroups_hugetlb.t validation/linux_cgroups_memory.t validation/linux_rootfs_propagation_shared.t validation/kill.t validation/create.t validation/poststart.t validation/linux_cgroups_network.t validation/poststop_fail.t validation/linux_readonly_paths.t validation/prestart_fail.t validation/hooks_stdin.t validation/default.t validation/linux_masked_paths.t validation/poststop.t validation/misc_props.t validation/prestart.t validation/poststart_fail.t validation/mounts.t validation/linux_cgroups_relative_pids.t validation/process_user.t validation/process.t validation/hooks.t validation/process_capabilities_fail.t validation/process_rlimits_fail.t validation/linux_cgroups_relative_cpus.t validation/process_rlimits.t validation/linux_cgroups_relative_blkio.t validation/linux_sysctl.t validation/linux_seccomp.t validation/linux_devices.t validation/start.t validation/linux_cgroups_pids.t validation/process_capabilities.t validation/process_oom_score_adj.t validation/linux_cgroups_relative_hugetlb.t validation/linux_cgroups_cpus.t validation/linux_cgroups_relative_memory.t validation/state.t validation/root_readonly_true.t validation/linux_cgroups_blkio.t validation/linux_rootfs_propagation_unbindable.t validation/delete.t validation/linux_cgroups_relative_network.t validation/hostname.t validation/killsig.t validation/linux_uid_mappings.t
validation/pidfile.t .failed to create the container
container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 56892210544640 to hugetlb.1GB.limit_in_bytes: open /sys/fs/cgroup/hugetlb/cgrouptest/hugetlb.1GB.limit_in_bytes: permission denied\\\"\""
exit status 1
validation/pidfile.t .................................. 1/1 315ms
validation/linux_cgroups_hugetlb.t .................... 0/1
not ok validation/linux_cgroups_hugetlb.t
timeout: 30000
file: validation/linux_cgroups_hugetlb.t
command: validation/linux_cgroups_hugetlb.t
args: []
stdio:
- 0
- pipe
- 2
cwd: /…/go/src/github.com/opencontainers/runtime-tools
exitCode: 1

validation/linux_cgroups_memory.t ..................... 9/9
validation/linux_rootfs_propagation_shared.t ...... 252/282
not ok shared root propagation exposes "/target348456609/mount892511628/example376408222"

Skipped: 29
/dev/null (default device) has unconfigured permissions
total ........................................... 4381/4962


4381 passing (1m)
567 pending
14 failing

make: *** [Makefile:44: localvalidation] Error 1
```

You can also run an individual test executable directly:

```console
$ RUNTIME=runc validation/default.t
TAP version 13
ok 1 - has expected hostname
---
{
"actual": "mrsdalloway",
"expected": "mrsdalloway"
}
...
ok 287 # SKIP linux.gidMappings not set
1..287
```

If you cannot install node-tap, you can probably run the test suite with another [TAP consumer][tap-consumers].
For example, with [`prove`][prove]:

```console
$ sudo make TAP='prove -Q -j9' RUNTIME=runc VALIDATION_TESTS=validation/pidfile.t localvalidation
RUNTIME=runc prove -Q -j9 validation/pidfile.t
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.01 usr 0.01 sys + 0.03 cusr 0.03 csys = 0.08 CPU)
Result: PASS
```
Use the [runtime validation suite](doc/runtime-compliance-testing.md).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doc -> docs

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@q384566678 Good catch. Fixed. Thanks!


[bundle]: https://github.com/opencontainers/runtime-spec/blob/master/bundle.md
[config.json]: https://github.com/opencontainers/runtime-spec/blob/master/config.md
[debian-node-tap]: https://packages.debian.org/stretch/node-tap
[debian-nodejs]: https://packages.debian.org/stretch/nodejs
[gentoo-nodejs]: https://packages.gentoo.org/packages/net-libs/nodejs
[node-tap]: http://www.node-tap.org/
[npm]: https://www.npmjs.com/
[prove]: http://search.cpan.org/~leont/Test-Harness-3.39/bin/prove
[runC]: https://github.com/opencontainers/runc
[runtime-spec]: https://github.com/opencontainers/runtime-spec
[tap-consumers]: https://testanything.org/consumers.html

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if we have a runtime-tools policy on this point, but runtime-spec prefers a blank line between the non-man-page and man-page references.

[generate.1]: man/oci-runtime-tool-generate.1.md
[validate.1]: man/oci-runtime-tool-validate.1.md
158 changes: 158 additions & 0 deletions docs/runtime-compliance-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,162 @@ In order to be tested for [compliance][], runtimes MUST support at least one of

* Version 1.0.1 of the [OCI Runtime Command Line Interface](command-line-interface.md).

## Running the runtime validation suite from source

The runtime validation suite uses [node-tap][], which is packaged for some distributions (for example, it is in [Debian's `node-tap` package][debian-node-tap]).
If your distribution does not package node-tap, you can install [npm][] (for example, from [Gentoo's `nodejs` package][gentoo-nodejs]) and use it:

```console
$ npm install tap
```

Build the validation executables:

```console
$ make runtimetest validation-executables
```

Runtime validation currently [only supports](docs/runtime-compliance-testing.md) the [OCI Runtime Command Line Interface](doc/command-line-interface.md).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These relative links (and some later ones in this file) need to be updated now that you've moved the text under docs/.

Also, this information is generic for runtime-validation; it's not specific to the “From source” approach. Can you shift it (and some of the text from the next paragraph) up into the “Running the runtime validation suite” section?

If we add support for alternative APIs in the future, runtime validation will gain an option to select the desired runtime API.
For the command line interface, the `RUNTIME` option selects the runtime command (`funC` in the [OCI Runtime Command Line Interface](doc/command-line-interface.md)).

```
$ sudo make RUNTIME=runc localvalidation
RUNTIME=runc tap validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t
validation/linux_rootfs_propagation_shared.t ........ 18/19
not ok rootfs propagation
error: 'rootfs should be shared, but not'

validation/create.t ................................... 4/4
validation/default.t ................................ 19/19
validation/linux_readonly_paths.t ................... 19/19
validation/linux_masked_paths.t ..................... 18/19
not ok masked paths
error: /masktest should not be readable

validation/mounts.t ................................... 0/1
Skipped: 1
TODO: mounts generation options have not been implemented

validation/process.t ................................ 19/19
validation/root_readonly_false.t .................... 19/19
validation/linux_sysctl.t ........................... 19/19
validation/linux_devices.t .......................... 19/19
validation/linux_gid_mappings.t ..................... 18/19
not ok gid mappings

validation/process_oom_score_adj.t .................. 19/19
validation/process_capabilities.t ................... 19/19
validation/process_rlimits.t ........................ 19/19
validation/root_readonly_true.t ...................failed to create the container
rootfsPropagation=unbindable is not supported
exit status 1
validation/root_readonly_true.t ..................... 19/19
validation/linux_rootfs_propagation_unbindable.t ...... 0/1
not ok validation/linux_rootfs_propagation_unbindable.t
timeout: 30000
file: validation/linux_rootfs_propagation_unbindable.t
command: validation/linux_rootfs_propagation_unbindable.t
args: []
stdio:
- 0
- pipe
- 2
cwd: /…/go/src/github.com/opencontainers/runtime-tools
exitCode: 1

validation/hostname.t ...................failed to create the container
User namespace mappings specified, but USER namespace isn't enabled in the config
exit status 1
validation/hostname.t ............................... 19/19
validation/linux_uid_mappings.t ....................... 0/1
not ok validation/linux_uid_mappings.t
timeout: 30000
file: validation/linux_uid_mappings.t
command: validation/linux_uid_mappings.t
args: []
stdio:
- 0
- pipe
- 2
cwd: /…/go/src/github.com/opencontainers/runtime-tools
exitCode: 1

total ............................................. 267/273


267 passing (31s)
1 pending
5 failing

make: *** [Makefile:43: localvalidation] Error 1
```

You can also run an individual test executable directly:

```console
$ RUNTIME=runc validation/default.t
TAP version 13
ok 1 - root filesystem
ok 2 - hostname
ok 3 - process
ok 4 - mounts
ok 5 - user
ok 6 - rlimits
ok 7 - capabilities
ok 8 - default symlinks
ok 9 - default file system
ok 10 - default devices
ok 11 - linux devices
ok 12 - linux process
ok 13 - masked paths
ok 14 - oom score adj
ok 15 - read only paths
ok 16 - rootfs propagation
ok 17 - sysctls
ok 18 - uid mappings
ok 19 - gid mappings
1..19
```

If you cannot install node-tap, you can probably run the test suite with another [TAP consumer][tap-consumers].
For example, with [`prove`][prove]:

```console
$ sudo make TAP='prove -Q -j9' RUNTIME=runc localvalidation
RUNTIME=runc prove -Q -j9 validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t
failed to create the container
rootfsPropagation=unbindable is not supported
exit status 1
failed to create the container
User namespace mappings specified, but USER namespace isn't enabled in the config
exit status 1

Test Summary Report
-------------------
validation/linux_rootfs_propagation_shared.t (Wstat: 0 Tests: 19 Failed: 1)
Failed test: 16
validation/linux_masked_paths.t (Wstat: 0 Tests: 19 Failed: 1)
Failed test: 13
validation/linux_rootfs_propagation_unbindable.t (Wstat: 256 Tests: 0 Failed: 0)
Non-zero exit status: 1
Parse errors: No plan found in TAP output
validation/linux_uid_mappings.t (Wstat: 256 Tests: 0 Failed: 0)
Non-zero exit status: 1
Parse errors: No plan found in TAP output
validation/linux_gid_mappings.t (Wstat: 0 Tests: 19 Failed: 1)
Failed test: 19
Files=18, Tests=271, 6 wallclock secs ( 0.06 usr 0.01 sys + 0.59 cusr 0.24 csys = 0.90 CPU)
Result: FAIL
make: *** [Makefile:43: localvalidation] Error 1
```


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You only need one blank line here.

[compliance]: https://github.com/opencontainers/runtime-spec/blob/v1.0.1/spec.md
[debian-node-tap]: https://packages.debian.org/stretch/node-tap
[debian-nodejs]: https://packages.debian.org/stretch/nodejs
[gentoo-nodejs]: https://packages.gentoo.org/packages/net-libs/nodejs
[node-tap]: http://www.node-tap.org/
[npm]: https://www.npmjs.com/
[prove]: http://search.cpan.org/~leont/Test-Harness-3.39/bin/prove
[tap-consumers]: https://testanything.org/consumers.html