Skip to content

Create Image Tools Interface #26

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
RobDolinMS wants to merge 3 commits into from
Closed

Create Image Tools Interface #26

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0d64598
Create Image Tools Interface
RobDolinMS Mar 20, 2017
b0268a6
[Image Tools Interface] Add Image Spec version
RobDolinMS Apr 13, 2017
511b4ac
Merge pull request #27 from opencontainers/RobDolinMS-image-spec-version
RobDolinMS Apr 13, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions image-tools-interface.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Image Tools Interface

The below document describes an interface for the OCI Image Tools project as desired by the OCI Certification Program working group (Cert WG.)

Goals:
* An engineer can verify a container image complies with the OCI Image Spec (https://github.com/opencontainers/image-spec/releases)
Copy link

@wking wking Apr 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trailing whitespace on this line (and a few others):

$ git diff --check origin/master..origin/pr/26
image-tools-interface.md:6: trailing whitespace.
+* An engineer can verify a container image complies with the OCI Image Spec (https://github.com/opencontainers/image-spec/releases) 
image-tools-interface.md:11: trailing whitespace.
+* (OPTIONAL): Which version of the OCI Image Spec to validate against 
image-tools-interface.md:25: trailing whitespace.
+INPUT: 
image-tools-interface.md:28: trailing whitespace.
+OUTPUT (default): 
image-tools-interface.md:31: trailing whitespace.
+Case 12: FAILED: Image using reserved field key: https://github.com/opencontainers/image-spec/blob/master/descriptor.md#reserved 
image-tools-interface.md:32: trailing whitespace.
+Results: 
image-tools-interface.md:48: trailing whitespace.
+Results:


Inputs:
* (REQUIRED): Path or reference to a container image
Copy link

@alban alban Mar 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If distribution is out of scope of image-spec, what would a "reference" be? A https:// URL to the OCI Image Layout directory? And the software would download ${URL}/index.json and other files as required by the image layout content?

* (OPTIONAL): If the user would like verbose output
* (OPTIONAL): Which version of the OCI Image Spec to validate against
(IF the OCI Image Tool supports multiple OCI Image Spec versions; Default to highest supported)
Copy link

@wking wking Apr 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“IF” → “If”?


Outputs:
* Number of cases run
* Number of cases passed
* Number of cases failed
* List of cases failed
* Version of OCI Image Tools
Copy link
Member

@vbatts vbatts Apr 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and maybe git commit of the tool?
and version of the OCI spec tested against.
And the timestamp the validation just ran.
Also, presumably this would need to output the digests of the objects tested.

With this data points in a structured output (i.e. xml, json, etc) then it could be more deterministically used for badges, etc.

Copy link

@wking wking Apr 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and maybe git commit of the tool?

I like this, but note that in a similar context the OCI consensus (as voiced by @duglin here) was that a consistent API for extracting the implementation version was not required for interop.

And the timestamp the validation just ran.

Does this matter? Hopefully the version of the test suite/tools and the digest(s) of the tested object would be sufficient without recourse to the validation timestamp. Higher-level tooling can always add a timestamp if it feels it would be useful.

Also, presumably this would need to output the digests of the objects tested.

This is tricky for stuff outside of CAS. For example, if validation was pointed at an image-layout via a HTTP URL, or at a directory, what digest hash do you show for complaining about “I couldn't find index.json there”?

Copy link
Contributor Author

@RobDolinMS RobDolinMS Apr 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replying to comments from @vbatts and @wking:

and maybe git commit of the tool?

I was thinking that 'version of the OCI Image Tools' would be sufficient since only released versions of the tools will be relevant for image validation.

and version of the OCI spec tested against.

I didn't think of this, but it makes sense. I'll update the PR.

And the timestamp the validation just ran.
I'm not sure this is needed since a released version of the OCI Image Tool, a released version of the OCI Image Spec, and the target image should have the same output every time.

Also, presumably this would need to output the digests of the objects tested.

@vbatts Would you mind suggesting what this might look like?

Copy link
Contributor Author

@RobDolinMS RobDolinMS Apr 13, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't think of [version of the OCI spec tested against], but it makes sense. I'll update the PR.

PR is updated at lines 11, 12, 20, 30, and 40. /cc @vbatts @wking

* Version of OCI Image Spec
* (IF VERBOSE) List of cases passed

## Possible Example

INPUT:
`oci-image-tools image validate <image>`

OUTPUT (default):
```
OCI Image Tools vX.Y.Z validating <image> with OCI Image Spec vA.B.C:
Case 12: FAILED: Image using reserved field key: https://github.com/opencontainers/image-spec/blob/master/descriptor.md#reserved
Results:
* Ran: 24 / 24 (100%) validation cases
* Passed: 23 / 24 (96%)
* Failed: 1 / 24 (4%)
```

OUTPUT (verbose):
```
OCI Image Tools vX.Y.Z validating <image> with OCI Image Spec vA.B.C:
Copy link

@stevvooe stevvooe Apr 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be good to look at rust's compiler error documentation for the items that might be required here for each error. As this is, it is a little hard to parse.

Let's also take a peak at Go's output:

=== RUN   TestCopyDirectory
--- PASS: TestCopyDirectory (0.00s)
=== RUN   TestSimpleDiff
--- PASS: TestSimpleDiff (0.00s)
=== RUN   TestDirectoryReplace
--- PASS: TestDirectoryReplace (0.00s)
=== RUN   TestRemoveDirectoryTree
--- PASS: TestRemoveDirectoryTree (0.00s)
=== RUN   TestFileReplace
--- PASS: TestFileReplace (0.00s)
=== RUN   TestUpdateWithSameTime
--- PASS: TestUpdateWithSameTime (0.00s)
=== RUN   TestBaseDirectoryChanges
--- PASS: TestBaseDirectoryChanges (0.00s)
PASS
ok  	github.com/containerd/containerd/fs	0.007s

Each line is marked as "starting test" and "ending test" (=== and --- respectively), the we get a parseable package name at the end. Timing is less important for validation, so we don't need both, but we need a way to differentiate a result from other information.

Something like this might work:

* TestName1: PASSED
* TestNameFailure: FAILED https://github.com/opencontainers/image-spec/blob/master/descriptor.md#reserved
... Information about failure.
* TestName2: PASSED
# Results: 23 / 24 (96%) passed

Notice that the status and result lines are marked with a leading character, leaving the other output to be expressive. There may be better characters than * for marking these lines, but we can see how we mark what is consumable by a leading character.

Copy link
Member

@cyphar cyphar Apr 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could also use TAP rather than making our own output format.

Case 1: PASSED: …
Case 2: PASSED: …
…
Case 12: FAILED: Image using reserved field key: https://github.com/opencontainers/image-spec/blob/master/descriptor.md#reserved
Case 13: PASSED: …
…
Case 24: PASSED: …
Results:
* Ran: 24 / 24 (100%) validation cases
* Passed: 23 / 24 (96%)
* Failed: 1 / 24 (4%)
```