diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..bee8a64 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +__pycache__ diff --git a/DESIGN-SPEC.md b/DESIGN-SPEC.md deleted file mode 100644 index 93e7c8a..0000000 --- a/DESIGN-SPEC.md +++ /dev/null @@ -1 +0,0 @@ -# OCI Image Discovery diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..5d725a0 --- /dev/null +++ b/Makefile @@ -0,0 +1,19 @@ +# Copyright 2017 oci-discovery contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +test: + python3 -m unittest discover + +test-debug: + DEBUG=1 python3 -m unittest discover -v diff --git a/README.md b/README.md index c4f60f5..14acfaf 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,262 @@ -# README +# OCI Image Discovery Specifications -This repository is to present the [discovery specification](DESIGN-SPEC.md) OCI image discovery functionality, as part of [image-spec](https://github.com/opencontainers/image-spec). +This repository contains the [OCI Ref-engine Discovery specification](ref-engine-discovery.md) and related specifications as an extention to the [image specification][image-spec]: -For OCI image distibution is a complex and wide system, and much implementation should will done, [discovery specification](DESIGN-SPEC.md) document just be separated from that, to achieve only consensual image discovery protocol. +* [Host-Based Image Names](host-based-image-names.md) + There is a [Python 3][python3] implementation in [`oci_discovery.host_based_image_names`](oci_discovery/host_based_image_names). +* [OCI Ref-engine Discovery](ref-engine-discovery.md). + There is a Python 3 implementation in [`oci_discovery.ref_engine_discovery`](oci_discovery/ref_engine_discovery). +* [OCI Index Template Protocol](index-template.md) + There is a Python 3 implementation in [`oci_discovery.ref_engine.oci_index_template`](oci_discovery/ref_engine/oci_index_template). +* [OCI CAS Template Protocol](cas-template.md) -The strategies in this document refer to existing great open implementations. +This repository also contains registries for ref- and CAS-engine protocols: -* [ABD](https://github.com/appc/abd/blob/master/abd.md) +* [Ref-Engine Protocols](ref-engine-prococols.md). + There is a Python 3 implementation in [`oci_discovery.ref_engine.CONSTRUCTORS`](oci_discovery/ref_engine/__init__.py). +* [CAS-Engine Protocols](cas-engine-protocols.md). -* [App Container Image Discovery](https://github.com/appc/spec/blob/v0.8.10/spec/discovery.md) +The strategies in these specifications are inspired by some previous implementations: +* [ABD](https://github.com/appc/abd/blob/master/abd.md) +* [App Container Image Discovery](https://github.com/appc/spec/blob/v0.8.10/spec/discovery.md) * [parcel](https://github.com/cyphar/parcel) + +## Using the Python 3 ref-engine discovery tool + +The individual components are usable as libraries, but the ref-engine discovery implementation can also be used from the command line: + +``` +$ python -m oci_discovery.ref_engine_discovery -l debug example.com/app#1.0 2>/tmp/log +{ + "example.com/app#1.0": { + "roots": [ + { + "annotations": { + "org.opencontainers.image.ref.name": "1.0" + }, + "casEngines": [ + { + "protocol": "oci-cas-template-v1", + "uri": "https://a.example.com/cas/{algorithm}/{encoded:2}/{encoded}" + } + ], + "digest": "sha256:e9770a03fbdccdd4632895151a93f9af58bbe2c91fdfaaf73160648d250e6ec3", + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "platform": { + "architecture": "ppc64le", + "os": "linux" + }, + "size": 799 + } + ] + } +} +$ cat /tmp/log +DEBUG:oci_discovery.ref_engine_discovery:discovering ref engines via https://example.com/.well-known/oci-host-ref-engines +WARNING:oci_discovery.ref_engine_discovery:failed to fetch https://example.com/.well-known/oci-host-ref-engines () +DEBUG:oci_discovery.ref_engine_discovery:discovering ref engines via http://example.com/.well-known/oci-host-ref-engines +DEBUG:oci_discovery.ref_engine_discovery:received ref-engine discovery object: +{'refEngines': [{'protocol': 'oci-index-template-v1', + 'uri': 'http://{host}/oci-index/{path}'}]} +DEBUG:oci_discovery.ref_engine.oci_index_template:fetching an OCI index for example.com/app#1.0 from http://example.com/oci-index/app +DEBUG:oci_discovery.ref_engine.oci_index_template:received OCI index object: +{'manifests': [{'annotations': {'org.opencontainers.image.ref.name': '1.0'}, + 'casEngines': [{'protocol': 'oci-cas-template-v1', + 'uri': 'https://a.example.com/cas/{algorithm}/{encoded:2}/{encoded}'}], + 'digest': 'sha256:e9770a03fbdccdd4632895151a93f9af58bbe2c91fdfaaf73160648d250e6ec3', + 'mediaType': 'application/vnd.oci.image.manifest.v1+json', + 'platform': {'architecture': 'ppc64le', 'os': 'linux'}, + 'size': 799}, + {'annotations': {'org.freedesktop.specifications.metainfo.type': 'AppStream', + 'org.freedesktop.specifications.metainfo.version': '1.0'}, + 'casEngines': [{'protocol': 'oci-cas-template-v1', + 'uri': 'https://b.example.com/cas/{algorithm}/{encoded}'}], + 'digest': 'sha256:b3d63d132d21c3ff4c35a061adf23cf43da8ae054247e32faa95494d904a007e', + 'mediaType': 'application/xml', + 'size': 7143}], + 'schemaVersion': 2} +``` + +Consumers who are trusting images based on the ref-engine discovery and ref-engine servers are encouraged to use `--https-only`. + +Consumers who are trusting images based on a property of the Merkle tree (e.g. [like this][signed-name-assertions]) can safely perform ref-engine discovery and ref-resolution over HTTP, although they may still want to use `--https-only` to protect from sniffers. + +## Example: Serving everything from one Nginx server + +Publishers who intend to serve discoverable images via the protocols in this repository, but who only want to serve static content can use [Nginx][] with a configuration like: + +``` +events { + worker_connections 1024; +} + +http { + # you may need to configure these if you lack write access to the + # default locations, depending on which features are compiled into + # your Nginx. + client_body_temp_path /some/where/client_temp; + proxy_temp_path /some/where/proxy_temp; + fastcgi_temp_path /some/where/fastcgi_temp; + scgi_temp_path /some/where/scgi_temp; + uwsgi_temp_path /some/where/uwsgi_temp; + + server { + listen 80; + listen [::]:80; + server_name example.com; + + location / { + return 301 https://$host$request_uri; + } + } + + server { + listen 443 ssl; + listen [::]:443 ssl; + server_name example.com; + + ssl_certificate /etc/ssl/example.com/fullchain.pem; + ssl_certificate_key /etc/ssl/example.com/privkey.pem; + + root /srv/example.com; + + location /.well-known/oci-host-ref-engines { + types {} + default_type application/vnd.oci.ref-engines.v1+json; + charset utf-8; + charset_types *; + } + + location /oci-index { + types {} + default_type application/vnd.oci.image.index.v1+json; + charset utf-8; + charset_types *; + } + } +} +``` + +Then in `/srv/example.com/.well-known/oci-host-ref-engines`, the following [ref-engines object](ref-engine-discovery.md#ref-engines-objects): + +```json +{ + "refEngines": [ + { + "protocol": "oci-index-template-v1", + "uri": "https://{host}/oci-index/{path}" + } + ] +} +``` + +With that pattern, consumers will attempt to resolve image names matching the `example.com/app#…` family of [host-based image names](host-based-image-names.md) via an [OCI Index Template](index-template.md) ref engine at `https://example.com/oci-index/app`. +Supply that by adding `application/vnd.oci.image.index.v1+json` content to `/srv/example.com/oci-index/app`: + +```json +{ + "schemaVersion": 2, + "manifests": [ + { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "size": 799, + "digest": "sha256:e9770a03fbdccdd4632895151a93f9af58bbe2c91fdfaaf73160648d250e6ec3", + "platform": { + "architecture": "ppc64le", + "os": "linux" + }, + "annotations": { + "org.opencontainers.image.ref.name": "1.0" + }, + "casEngines": [ + { + "protocol": "oci-cas-template-v1", + "uri": "https://example.com/oci-cas/{algorithm}/{encoded:2}/{encoded}" + } + ] + } + ] +} +``` + +The `org.opencontainers.image.ref.name` value assumes consumers will only be attempting to match the `fragment` and not the full image name; image-spec does not currently provide guidance on this point. + +Supply the blobs under `/srv/example.com/oci-cas`. For example, `/srv/example.com/oci-cas/sha256/e9/e9770a03fbdccdd4632895151a93f9af58bbe2c91fdfaaf73160648d250e6ec3` would contain: + +```json +{ + "schemaVersion": 2, + "config": { + "mediaType": "application/vnd.oci.image.config.v1+json", + "size": 7023, + "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7" + }, + "layers": [ + { + "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", + "size": 32654, + "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f" + }, + { + "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", + "size": 16724, + "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b" + }, + { + "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", + "size": 73109, + "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736" + } + ] +} +``` + +It would be more conformant [if that content was canonical JSON][image-spec-canonical-json], but I've added newlines and indents to make the example more readable. + +To publish additional images matching the `example.com/app#…` family of [host-based image names](host-based-image-names.md), add their entries to `/srv/example.com/oci-index/app`'s `manifests` array. +To publish additional images matching new families (e.g. `example.com/other-app#…`), add their entries to new `/srv/example.com/oci-index/` indexes (e.g. `/srv/example.com/oci-index/other-app`). +All the CAS blobs can go in the same bucket under `/srv/example.com/oci-cas`, although if you want you can adjust the `casEngines` entries and keep CAS blobs in different buckets. + +## Example: Serving OCI layouts from Nginx + +As an alternative to the [previous example](#example-serving-everything-from-one-nginx-server), you can bucket your CAS blobs by serving [OCI layouts][layout] directly. +If your layout `index.json` are not setting `casEngines` and you are unwilling to update them to do so, you can [set `casEngines` in you ref-engines object](ref-engine-discovery.md#ref-engines-objects) at `/srv/example.com/.well-known/oci-host-ref-engines`: + +```json +{ + "refEngines": [ + { + "protocol": "oci-index-template-v1", + "uri": "https://{host}/oci-image/{path}/index.json" + } + ], + "casEngines": [ + { + "protocol": "oci-cas-template-v1", + "uri": "https://example.com/oci-image/{path}/blobs/{algorithm}/{encoded}" + } + ] +} +``` + +Then copy your [layout directories][layout] under `/srv/example.com/oci-image/{path}` to deploy them. + +The Nginx config from the [previous example](#example-serving-everything-from-one-nginx-server) would need an adjusted [`location`][location] for the index media type: + +``` +location ~ ^/oci-image/.*/index.json$ { + types {} + default_type application/vnd.oci.image.index.v1+json; + charset utf-8; + charset_types *; +} +``` + +[image-spec]: https://github.com/opencontainers/image-spec +[image-spec-canonical-json]: https://github.com/opencontainers/image-spec/blob/v1.0.0/considerations.md#json +[layout]: https://github.com/opencontainers/image-spec/blob/v1.0.0/image-layout.md +[location]: http://nginx.org/en/docs/http/ngx_http_core_module.html#location +[Nginx]: https://nginx.org/ +[python3]: https://docs.python.org/3/ +[signed-name-assertions]: https://github.com/opencontainers/image-spec/issues/176 diff --git a/cas-engine-protocols.md b/cas-engine-protocols.md new file mode 100644 index 0000000..9cf5d16 --- /dev/null +++ b/cas-engine-protocols.md @@ -0,0 +1,14 @@ +# CAS-Engine Protocols + +There are many possible [CAS][] engine protocols. +Having identifiers for the protocols provides a standardized way to share structured connection information. +Consumers can then prefer CAS engines which implement their favorite protocol and use the appropriate API to connect to them. + +This section registers known protocol identifiers and maps them to their specification. +Anyone may submit new CAS-engine protocol identifiers for registration. + +| Protocol identifier | Specification | +|-------------------------|---------------------------------------------------------| +| `oci-cas-template-v1` | [OCI CAS template protocol, version 1](cas-template.md) | + +[CAS]: https://en.wikipedia.org/wiki/Content-addressable_storage diff --git a/cas-template.md b/cas-template.md new file mode 100644 index 0000000..ca15775 --- /dev/null +++ b/cas-template.md @@ -0,0 +1,39 @@ +# OCI CAS Template Protocol + +This is version 1 of this specification. + +The CAS-template protocol is configured via a single [URI Template][rfc6570]. +When configured via a [`casEngines` entry](ref-engine-discovery.md#ref-engines-objects), the `uri` property MUST be set, and its value is the URI Template. + +For a given blob digest, consumers MUST provide at least the following variables: + +* `digest`, matching `digest` in the [`digest` rule][digest]. +* `algorithm`, matching `algorithm` in the `digest` rule. +* `encoded`, matching `encoded` in the `digest` rule. + +and expand the URI Template as defined in [RFC 6570 section 3][rfc6570-s3]. + +## Example + +An example [`casEngines` entry](ref-engine-discovery.md#ref-engines-objects) using the [registered `oci-cas-template-v1` protocol identifier](cas-engine-protocols.md) is: + +```json +{ + "protocol": "oci-cas-template-v1", + "uri": "https://a.example.com/cas/{algorithm}/{encoded:2}/{encoded}" +} +``` + +A digest like `sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` matches [`digest`][digest] with: + +* `sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` as `digest`, +* `sha256` as `algorithm`, and +* `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` as `encoded` + +so the expanded URI is: + + https://a.example.com/cas/sha256/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 + +[digest]: https://github.com/opencontainers/image-spec/blob/v1.0.0/descriptor.md#digests +[rfc6570]: https://tools.ietf.org/html/rfc6570 +[rfc6570-s3]: https://tools.ietf.org/html/rfc6570#section-3 diff --git a/host-based-image-names.md b/host-based-image-names.md new file mode 100644 index 0000000..c42434b --- /dev/null +++ b/host-based-image-names.md @@ -0,0 +1,34 @@ +# Host-Based Image Names + +This is version 1 of this specification. + +The [X.509 Public Key Infrastructure][X.509] provides a well-established mechanism for trusted namespacing of [domain names][rfc5890]. +Protocols interested in leveraging that infrastructure need to be able to extract domain names from image names. +This specification provides one approach for that extraction. + +This specification defines image names compatible with host names using the following [ABNF][]: + +```ABNF +host-based-image-name = host "/" path-rootless [ "#" fragment ] +``` + +where: + +* `host` is defined in [RFC 3986 section 3.2.2][rfc3986-s3.2.2]. + While IP addresses are valid host names, X.509 certificates usually assert ownership of one or more domain names and do not mention IP addresses. + Host-based image names SHOULD use host names that conform [RFC 1034's preferred name syntax][rfc1034-s3.5] as modified by [RFC 1123 section 2.1][rfc1123-s2.1]. +* `path-rootless` is defined in [RFC 3986 section 3.3][rfc3986-s3.3]. +* `fragment` is defined in [RFC 3986 section 3.5][rfc3986-s3.5]. + +Implementations MAY accept other names, for example, by creating a default `host` for names that match `segment-nz` (defined in [RFC 3986 section 3.3][rfc3986-s3.3]). + +Names which are not supported for `host-based-image-name` will not be able to use protocols that rely on this rule, although they may use other protocols. + +[ABNF]: https://tools.ietf.org/html/rfc5234 +[rfc1034-s3.5]: https://tools.ietf.org/html/rfc1034#section-3.5 +[rfc1123-s2.1]: https://tools.ietf.org/html/rfc1123#section-2 +[rfc3986-s3.2.2]: https://tools.ietf.org/html/rfc3986#section-3.2.2 +[rfc3986-s3.3]: https://tools.ietf.org/html/rfc3986#section-3.3 +[rfc3986-s3.5]: https://tools.ietf.org/html/rfc3986#section-3.5 +[rfc5890]: https://tools.ietf.org/html/rfc5890 +[X.509]: https://tools.ietf.org/html/rfc5280 diff --git a/index-template.md b/index-template.md new file mode 100644 index 0000000..c91cf0b --- /dev/null +++ b/index-template.md @@ -0,0 +1,94 @@ +# OCI Index Template Protocol + +This is version 1 of this specification. + +The index-template protocol is configured via a single [URI Template][rfc6570]. +When configured via a [`refEngines` entry](ref-engine-discovery.md#ref-engines-objects), the `uri` property MUST be set, and its value is the URI Template. + +Consumers MUST provide at least the following variables: + +* `name`, matching `host-based-image-name` in the [`host-based-image-name` rule](host-based-image-names.md). +* `host`, matching `host` in the `host-based-image-name` rule. +* `path`, matching `path-rootless` in the `host-based-image-name` rule. +* `fragment`, matching `fragment` in the `host-based-image-name` rule. + If `fragment` was not provided in the image name, it defaults to an empty string. + +and expand the URI Template as defined in [RFC 6570 section 3][rfc6570-s3]. + +The server providing the expanded URI MUST support requests for media type [`application/vnd.oci.image.index.v1+json`][index]. +Servers MAY support other media types using HTTP content negotiation, as described in [RFC 7231 section 3.4][rfc7231-s3.4] (which is [also supported over HTTP/2][rfc7540-s8]). + +Consumers retrieving `application/vnd.oci.image.index.v1+json` SHOULD process it like a [layout's `index.json`][index.json], respecting [`org.opencontainers.image.ref.name` and other annotations which are recommended for `index.json`][annotations]. + +## Example + +An example [`refEngines` entry](ref-engine-discovery.md#ref-engines-objects) using the [registered `oci-index-template-v1` protocol identifier](ref-engine-protocols.md) is: + +```json +{ + "protocol": "oci-index-template-v1", + "uri": "https://{host}/ref/{host}/{path}" +} +``` + +An image name like `a.b.example.com/c/d#1.0` matches [`host-based-image-name`](host-based-image-names.md) with `a.b.example.com` as `host`, `c/d` as `path-rootless`, and `1.0` as `fragment` so the expanded URI is: + + https://a.b.example.com/ref/a.b.example.com/c/d + +Retrieving that URI (with a pretend result, since [`example.com` is reserved][rfc2606-s3]): + +``` +$ curl -H 'Accept: application/vnd.oci.image.index.v1+json' https://a.b.example.com/ref/a.b.example.com/c/d +{ + "schemaVersion": 2, + "manifests": [ + { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "size": 799, + "digest": "sha256:e9770a03fbdccdd4632895151a93f9af58bbe2c91fdfaaf73160648d250e6ec3", + "platform": { + "architecture": "ppc64le", + "os": "linux" + }, + "annotations": { + "org.opencontainers.image.ref.name": "1.0" + }, + "casEngines": [ + { + "protocol": "oci-cas-template-v1", + "uri": "https://a.example.com/cas/{algorithm}/{encoded:2}/{encoded}" + } + ] + }, + { + "mediaType": "application/xml", + "size": 7143, + "digest": "sha256:b3d63d132d21c3ff4c35a061adf23cf43da8ae054247e32faa95494d904a007e", + "annotations": { + "org.freedesktop.specifications.metainfo.version": "1.0", + "org.freedesktop.specifications.metainfo.type": "AppStream" + }, + "casEngines": [ + { + "protocol": "oci-cas-template-v1", + "uri": "https://b.example.com/cas/{algorithm}/{encoded}" + } + ] + } + ] +} +``` + +The [`oci-cas-template-v1` protocol](cas-template.md) is [registered](cas-engine-protocols.md). + +Deciding whether to look for `1.0` (the `fragment`) or the full `a.b.example.com/c/d#1.0` name is left as an exercise for the reader, as is switching based on `platform` entries or [chosing between multiple entries with the same name][duplicate-name-resolution]. + +[annotations]: https://github.com/opencontainers/image-spec/blob/v1.0.0/annotations.md#pre-defined-annotation-keys +[duplicate-name-resolution]: https://github.com/opencontainers/image-spec/issues/588#event-1080723646 +[index]: https://github.com/opencontainers/image-spec/blob/v1.0.0/image-index.md +[index.json]: https://github.com/opencontainers/image-spec/blob/v1.0.0/image-layout.md#indexjson-file +[rfc2606-s3]: https://tools.ietf.org/html/rfc2606#section-3 +[rfc6570]: https://tools.ietf.org/html/rfc6570 +[rfc6570-s3]: https://tools.ietf.org/html/rfc6570#section-3 +[rfc7231-s3.4]: https://tools.ietf.org/html/rfc7231#section-3.4 +[rfc7540-s8]: https://tools.ietf.org/html/rfc7540#section-8 diff --git a/oci_discovery/__init__.py b/oci_discovery/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/oci_discovery/fetch_json/__init__.py b/oci_discovery/fetch_json/__init__.py new file mode 100644 index 0000000..713023a --- /dev/null +++ b/oci_discovery/fetch_json/__init__.py @@ -0,0 +1,39 @@ +# Copyright 2017 oci-discovery contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import json as _json +import urllib.request as _urllib_request + + +def fetch(uri, media_type='application/json'): + """Fetch a JSON resource.""" + response = _urllib_request.urlopen(uri) + content_type = response.headers.get_content_type() + if content_type != media_type: + raise ValueError( + '{} returned {}, not {}'.format(uri, content_type, media_type)) + body_bytes = response.read() + charset = response.headers.get_content_charset() + if charset is None: + raise ValueError('{} does not declare a charset'.format(uri)) + try: + body = body_bytes.decode(charset) + except ValueError as error: + raise ValueError( + '{} returned content which did not match the declared {} charset' + .format(uri, charset)) from error + try: + return _json.loads(body) + except ValueError as error: + raise ValueError('{} returned invalid JSON'.format(uri)) from error diff --git a/oci_discovery/fetch_json/test.py b/oci_discovery/fetch_json/test.py new file mode 100644 index 0000000..9f628f9 --- /dev/null +++ b/oci_discovery/fetch_json/test.py @@ -0,0 +1,118 @@ +# Copyright 2017 oci-discovery contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import email.message +import unittest +import unittest.mock + +from . import fetch + + +class HTTPResponse(object): + def __init__(self, code=200, body=None, headers=None): + self.code = code + self._body = body + self.headers = email.message.Message() + for key, value in headers.items(): + self.headers[key] = value + + def read(self): + return self._body or '' + + +class TestFetchJSON(unittest.TestCase): + def test_good(self): + for name, response, expected in [ + ( + 'empty object', + HTTPResponse( + body=b'{}', + headers={ + 'Content-Type': 'application/json; charset=UTF-8', + }, + ), + {}, + ), + ( + 'basic object', + HTTPResponse( + body=b'{"a": "b", "c": 1}', + headers={ + 'Content-Type': 'application/json; charset=UTF-8', + }, + ), + {'a': 'b', 'c': 1}, + ), + ]: + with self.subTest(name=name): + with unittest.mock.patch( + target='oci_discovery.fetch_json._urllib_request.urlopen', + return_value=response + ) as patch_context: + json = fetch(uri='https://example.com') + self.assertEqual(json, expected) + + def test_bad(self): + for name, response, error, regex in [ + ( + 'no charset', + HTTPResponse( + body=b'{}', + headers={ + 'Content-Type': 'application/json', + }, + ), + ValueError, + 'https://example.com does not declare a charset' + ), + ( + 'declared charset does not match body', + HTTPResponse( + body=b'\xff', + headers={ + 'Content-Type': 'application/json; charset=UTF-8', + }, + ), + ValueError, + 'https://example.com returned content which did not match the declared utf-8 charset' + ), + ( + 'invalid JSON', + HTTPResponse( + body=b'{', + headers={ + 'Content-Type': 'application/json; charset=UTF-8', + }, + ), + ValueError, + 'https://example.com returned invalid JSON' + ), + ( + 'unexpected media type', + HTTPResponse( + body=b'{}', + headers={ + 'Content-Type': 'text/plain; charset=UTF-8', + }, + ), + ValueError, + 'https://example.com returned text/plain, not application/json' + ), + ]: + with self.subTest(name=name): + with unittest.mock.patch( + target='oci_discovery.fetch_json._urllib_request.urlopen', + return_value=response): + self.assertRaisesRegex( + error, regex, fetch, 'https://example.com') diff --git a/oci_discovery/host_based_image_names/__init__.py b/oci_discovery/host_based_image_names/__init__.py new file mode 100644 index 0000000..5e764d8 --- /dev/null +++ b/oci_discovery/host_based_image_names/__init__.py @@ -0,0 +1,45 @@ +# Copyright 2017 oci-discovery contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import re as _re + + +# The normative ABNF is in host-based-image-names.md referencing rules +# from https://tools.ietf.org/html/rfc3986#appendix-A. This +# regular expression is too liberal, but will successfully match all +# valid host-based image names. +_UNRESERVED_NO_HYPHEN = 'a-zA-Z0-9._~' +_SUB_DELIMS = "!$&'()*+,;=" +_PCHAR = _UNRESERVED_NO_HYPHEN + '%' + _SUB_DELIMS + ':@' + '-' +_HOST_BASED_IMAGE_NAME_REGEX = _re.compile( + '^(?P[^/]+)' + '/' + '(?P[' + _PCHAR + ']+(/[' + _PCHAR + ']*)*)' + '(#(?P[/?' + _PCHAR + ']*))?$') + + +def parse(name): + """Parse a host-based image name. + + Following host-based-image-names.md. + """ + match = _HOST_BASED_IMAGE_NAME_REGEX.match(name) + if match is None: + raise ValueError( + '{!r} does not match the host-based-image-name pattern' + .format(name)) + groups = match.groupdict() + if groups['fragment'] is None: + groups['fragment'] = '' + return groups diff --git a/oci_discovery/host_based_image_names/test.py b/oci_discovery/host_based_image_names/test.py new file mode 100644 index 0000000..43e3f19 --- /dev/null +++ b/oci_discovery/host_based_image_names/test.py @@ -0,0 +1,56 @@ +# Copyright 2017 oci-discovery contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import unittest + +from . import parse + + +class TestImageNameParsing(unittest.TestCase): + def test_good(self): + for (name, expected) in [ + ('example.com/a', { + 'host': 'example.com', + 'path': 'a', + 'fragment': '', + }), + ('example.com/a/', { + 'host': 'example.com', + 'path': 'a/', + 'fragment': '', + }), + ('example.com/a/b', { + 'host': 'example.com', + 'path': 'a/b', + 'fragment': '', + }), + ('example.com/a/b#c', { + 'host': 'example.com', + 'path': 'a/b', + 'fragment': 'c', + }), + ]: + with self.subTest(name=name): + match = parse(name=name) + self.assertEqual(match, expected) + + def test_bad(self): + for name in [ + 'example.com', + '/', + 'example.com/', + 'example.com/#', + ]: + with self.subTest(name=name): + self.assertRaises(ValueError, parse, name) diff --git a/oci_discovery/ref_engine/__init__.py b/oci_discovery/ref_engine/__init__.py new file mode 100644 index 0000000..5600bc2 --- /dev/null +++ b/oci_discovery/ref_engine/__init__.py @@ -0,0 +1,38 @@ +# Copyright 2017 oci-discovery contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from . import dummy as _dummy +from . import oci_index_template as _oci_index_template + +# Registry for ref engines, based on ref-engine-protocols.md. +CONSTRUCTORS = { + '_dummy': _dummy.Engine, + 'oci-index-template-v1': _oci_index_template.Engine, +} + + +def new(protocol, **kwargs): + """Construct a new ref engine from a refEngines entry. + + The returned ref engine MUST provide a 'resolve' method, which + takes an image name as a 'name' argument and returns an iterable + of Merkle root objects. Merkle root objects may be of any type, + but JSON root objects SHOULD be represented as Python dicts. + """ + try: + constructor = CONSTRUCTORS[protocol] + except KeyError: + raise ValueError( + 'unsupported ref-engine protocol {!r}'.format(protocol)) + return constructor(**kwargs) diff --git a/oci_discovery/ref_engine/dummy.py b/oci_discovery/ref_engine/dummy.py new file mode 100644 index 0000000..ea10e30 --- /dev/null +++ b/oci_discovery/ref_engine/dummy.py @@ -0,0 +1,33 @@ +# Copyright 2017 oci-discovery contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import copy as _copy + + +class Engine(object): + """Dummy ref engine for testing. + + So we don't have to hit the network to test resolution. + """ + def __str__(self): + return '<{}.{} response={}>'.format( + self.__class__.__module__, + self.__class__.__name__, + self._response) + + def __init__(self, response): + self._response = response + + def resolve(self, name): + return _copy.deepcopy(self._response) diff --git a/oci_discovery/ref_engine/oci_index_template.py b/oci_discovery/ref_engine/oci_index_template.py new file mode 100644 index 0000000..37f0dc9 --- /dev/null +++ b/oci_discovery/ref_engine/oci_index_template.py @@ -0,0 +1,71 @@ +# Copyright 2017 oci-discovery contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging as _logging +import pprint as _pprint + +from .. import fetch_json as _fetch_json +from .. import host_based_image_names as _host_based_image_names + + +_LOGGER = _logging.getLogger(__name__) + + +class Engine(object): + def __str__(self): + return '<{}.{} uri={}>'.format( + self.__class__.__module__, + self.__class__.__name__, + self.uri_template) + + def __init__(self, uri): + self.uri_template = uri + + def resolve(self, name): + name_parts = _host_based_image_names.parse(name=name) + try: + uri = self.uri_template.format(**name_parts) + except KeyError as error: + raise ValueError( + 'failed to format {}'.format(self.uri_template) + ) from error + _LOGGER.debug('fetching an OCI index for {} from {}'.format(name, uri)) + index = _fetch_json.fetch( + uri=uri, + media_type='application/vnd.oci.image.index.v1+json') + _LOGGER.debug('received OCI index object:\n{}'.format( + _pprint.pformat(index))) + if not isinstance(index, dict): + raise ValueError( + '{} claimed to return application/vnd.oci.image.index.v1+json, but actually returned {}' + .format(uri, index)) + if not isinstance(index.get('manifests', []), list): + raise ValueError( + '{} claimed to return application/vnd.oci.image.index.v1+json, but actually returned {}' + .format(uri, index)) + for entry in index.get('manifests', []): + if not isinstance(entry, dict): + raise ValueError( + '{} claimed to return application/vnd.oci.image.index.v1+json, but actually returned {}' + .format(uri, index)) + annotations = entry.get('annotations', {}) + if not isinstance(annotations, dict): + raise ValueError( + '{} claimed to return application/vnd.oci.image.index.v1+json, but actually returned {}' + .format(uri, index)) + entry_name = annotations.get( + 'org.opencontainers.image.ref.name', None) + if (name_parts['fragment'] == '' or + name_parts['fragment'] == entry_name): + yield entry diff --git a/oci_discovery/ref_engine/test_oci_index_template.py b/oci_discovery/ref_engine/test_oci_index_template.py new file mode 100644 index 0000000..f893269 --- /dev/null +++ b/oci_discovery/ref_engine/test_oci_index_template.py @@ -0,0 +1,137 @@ +# Copyright 2017 oci-discovery contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import unittest +import unittest.mock + +from . import oci_index_template + + +class TestEngine(unittest.TestCase): + def test_good(self): + for label, name, response, expected in [ + ( + 'empty fragment returns all entries', + 'example.com/a', + { + 'manifests': [ + { + 'entry': 'a', + }, + { + 'entry': 'b', + 'annotations': { + 'org.opencontainers.image.ref.name': '1.0', + }, + }, + ], + }, + [ + { + 'entry': 'a', + }, + { + 'entry': 'b', + 'annotations': { + 'org.opencontainers.image.ref.name': '1.0', + }, + }, + ], + ), + ( + 'nonempty fragment returns only matching entries', + 'example.com/a#1.0', + { + 'manifests': [ + { + 'entry': 'a', + }, + { + 'entry': 'b', + 'annotations': { + 'org.opencontainers.image.ref.name': '1.0', + }, + }, + ], + }, + [ + { + 'entry': 'b', + 'annotations': { + 'org.opencontainers.image.ref.name': '1.0', + }, + }, + ], + ), + ( + 'unmatched nonempty fragment returns no entries', + 'example.com/a#2.0', + { + 'manifests': [ + { + 'entry': 'a', + }, + { + 'entry': 'b', + 'annotations': { + 'org.opencontainers.image.ref.name': '1.0', + }, + }, + ], + }, + [], + ), + ]: + engine = oci_index_template.Engine(uri='https://example.com/index') + with self.subTest(label=label): + with unittest.mock.patch( + target='oci_discovery.ref_engine.oci_index_template._fetch_json.fetch', + return_value=response): + resolved = list(engine.resolve(name=name)) + self.assertEqual(resolved, expected) + + def test_bad(self): + for label, response, error, regex in [ + ( + 'index is not a JSON object', + [], + ValueError, + 'https://example.com/index claimed to return application/vnd.oci.image.index.v1\+json, but actually returned \[]', + ), + ( + 'manifests is not a JSON array', + {'manifests': {}}, + ValueError, + "https://example.com/index claimed to return application/vnd.oci.image.index.v1\+json, but actually returned \{'manifests': \{}}", + ), + ( + 'manifests contains a non-object', + {'manifests': [None]}, + ValueError, + "https://example.com/index claimed to return application/vnd.oci.image.index.v1\+json, but actually returned \{'manifests': \[None]}", + ), + ( + 'at least one manifests[].annotations is not a JSON object', + {'manifests': [{'annotations': None}]}, + ValueError, + "https://example.com/index claimed to return application/vnd.oci.image.index.v1\+json, but actually returned \{'manifests': \[\{'annotations': None}]}", + ), + ]: + engine = oci_index_template.Engine(uri='https://example.com/index') + with self.subTest(label=label): + with unittest.mock.patch( + target='oci_discovery.ref_engine.oci_index_template._fetch_json.fetch', + return_value=response): + generator = engine.resolve(name='example.com/a') + self.assertRaisesRegex(error, regex, list, generator) diff --git a/oci_discovery/ref_engine_discovery/__init__.py b/oci_discovery/ref_engine_discovery/__init__.py new file mode 100644 index 0000000..862c008 --- /dev/null +++ b/oci_discovery/ref_engine_discovery/__init__.py @@ -0,0 +1,77 @@ +# Copyright 2017 oci-discovery contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging as _logging +import pprint as _pprint +import urllib.error as _urllib_error + +from .. import fetch_json as _fetch_json +from .. import host_based_image_names as _host_based_image_names +from .. import ref_engine as _ref_engine +from . import ancestor_hosts as _ancestor_hosts + + +_LOGGER = _logging.getLogger(__name__) + + +def resolve(name, protocols=('https', 'http')): + """Resolve an image name to a Merkle root. + + Implementing ref-engine-discovery.md + """ + name_parts = _host_based_image_names.parse(name=name) + for protocol in protocols: + for host in _ancestor_hosts.ancestor_hosts(host=name_parts['host']): + uri = '{}://{}/.well-known/oci-host-ref-engines'.format( + protocol, host) + _LOGGER.debug('discovering ref engines via {}'.format(uri)) + try: + ref_engines_object = _fetch_json.fetch( + uri=uri, + media_type='application/vnd.oci.ref-engines.v1+json') + except _urllib_error.URLError as error: + _LOGGER.warning('failed to fetch {} ({})'.format(uri, error)) + continue + _LOGGER.debug('received ref-engine discovery object:\n{}'.format( + _pprint.pformat(ref_engines_object))) + if not isinstance(ref_engines_object, dict): + _LOGGER.warning( + '{} claimed to return application/vnd.oci.ref-engines.v1+json but actually returned {}' + .format(uri, ref_engines_object), + ) + continue + for ref_engine_object in ref_engines_object.get('refEngines', []): + try: + ref_engine = _ref_engine.new(**ref_engine_object) + except KeyError as error: + _LOGGER.warning(error) + continue + try: + roots = list(ref_engine.resolve(name=name)) + except _urllib_error.URLError as error: + _LOGGER.warning('failed to fetch {} ({})'.format( + error.geturl(), error)) + continue + except Exception as error: + _LOGGER.warning(error) + continue + if roots: + data = {'roots': roots} + if 'casEngines' in ref_engines_object: + data['casEngines'] = ref_engines_object['casEngines'] + return data + else: + _LOGGER.debug('{} returned no results for {}'.format( + ref_engine, name)) + raise ValueError('no Merkle root found for {!r}'.format(name)) diff --git a/oci_discovery/ref_engine_discovery/__main__.py b/oci_discovery/ref_engine_discovery/__main__.py new file mode 100644 index 0000000..df67f6a --- /dev/null +++ b/oci_discovery/ref_engine_discovery/__main__.py @@ -0,0 +1,65 @@ +# Copyright 2017 oci-discovery contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import json +import logging +import sys + +from . import resolve + + +logging.basicConfig() +log = logging.getLogger() +log.setLevel(logging.ERROR) + +parser = argparse.ArgumentParser( + description='Resolve image names via OCI Ref-engine Discovery.') +parser.add_argument( + 'names', metavar='NAME', type=str, nargs='+', + help='a host-based image name') +parser.add_argument( + '-l', '--log-level', + choices=['critical', 'error', 'warning', 'info', 'debug'], + help='Log verbosity. Defaults to {!r}.'.format( + logging.getLevelName(log.level).lower())) +parser.add_argument( + '--https-only', + action='store_const', + const=True, + help='Log verbosity. Defaults to {!r}.'.format( + logging.getLevelName(log.level).lower())) + +args = parser.parse_args() + +if args.log_level: + level = getattr(logging, args.log_level.upper()) + log.setLevel(level) + +protocols = ['https'] +if not args.https_only: + protocols.append('http') + +resolved = {} +for name in args.names: + try: + resolved[name] = resolve(name=name, protocols=protocols) + except ValueError as error: + log.error(error) +json.dump( + resolved, + sys.stdout, + indent=2, + sort_keys=True) +sys.stdout.write('\n') diff --git a/oci_discovery/ref_engine_discovery/ancestor_hosts.py b/oci_discovery/ref_engine_discovery/ancestor_hosts.py new file mode 100644 index 0000000..fa9d0eb --- /dev/null +++ b/oci_discovery/ref_engine_discovery/ancestor_hosts.py @@ -0,0 +1,38 @@ +# Copyright 2017 oci-discovery contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import re as _re + + +# Based on rules from https://tools.ietf.org/html/rfc3986#appendix-A. +_DEC_OCTET = '([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[05])' +_IP_V4_REGEXP = _re.compile( + '^' + _DEC_OCTET + '(\.' + _DEC_OCTET + '){3}$') + + +def ancestor_hosts(host): + """Iterate through a host and its DNS ancestors. + + Following ref-engine-discovery.md#images-associated-with-a-hosts-oci-host-ref-engines + """ + if host[0] == '[': + yield host + return # no ancestor domains for IP-literals + match = _IP_V4_REGEXP.match(host) + if match is not None: + yield host + return # no ancestor domains for IPv4 addresses + segments = host.split('.') + for i in range(len(segments) - 1): + yield '.'.join(segments[i:]) diff --git a/oci_discovery/ref_engine_discovery/test.py b/oci_discovery/ref_engine_discovery/test.py new file mode 100644 index 0000000..4c64efd --- /dev/null +++ b/oci_discovery/ref_engine_discovery/test.py @@ -0,0 +1,74 @@ +# Copyright 2017 oci-discovery contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +import os +import unittest +import unittest.mock + +from . import resolve + + +if 'DEBUG' in os.environ: + logging.basicConfig() + logging.getLogger().setLevel(logging.DEBUG) + + +class TestResolve(unittest.TestCase): + def test(self): + for label, name, response, expected in [ + ( + 'success', + 'example.com/a', + { + 'refEngines': [ + { + 'protocol': '_dummy', + 'response': [ + {'name': 'dummy Merkle root 1'}, + {'name': 'dummy Merkle root 2'}, + ], + } + ] + }, + { + 'roots': [ + {'name': 'dummy Merkle root 1'}, + {'name': 'dummy Merkle root 2'}, + ], + } + ), + ]: + with self.subTest(label=label): + with unittest.mock.patch( + target='oci_discovery.ref_engine_discovery._fetch_json.fetch', + return_value=response): + resolved = resolve(name=name) + self.assertEqual(resolved, expected) + + def test_bad(self): + for label, name, response, error, regex in [ + ( + 'ref-engine discovery not a JSON object', + 'example.com/a', + [], + ValueError, + "no Merkle root found for 'example.com/a'", + ), + ]: + with self.subTest(label=label): + with unittest.mock.patch( + target='oci_discovery.ref_engine_discovery._fetch_json.fetch', + return_value=response): + self.assertRaisesRegex(error, regex, resolve, name) diff --git a/oci_discovery/ref_engine_discovery/test_ancestor_hosts.py b/oci_discovery/ref_engine_discovery/test_ancestor_hosts.py new file mode 100644 index 0000000..64daee7 --- /dev/null +++ b/oci_discovery/ref_engine_discovery/test_ancestor_hosts.py @@ -0,0 +1,58 @@ +# Copyright 2017 oci-discovery contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import unittest + +from . import ancestor_hosts + + +class TestIPv4Detection(unittest.TestCase): + def test_ipv4(self): + for host, expected in [ + ('0.0.0.0', True), + ('9.0.0.0', True), + ('10.0.0.0', True), + ('99.0.0.0', True), + ('100.0.0.0', True), + ('199.0.0.0', True), + ('200.0.0.0', True), + ('249.0.0.0', True), + ('250.0.0.0', True), + ('255.0.0.0', True), + ('256.0.0.0', False), + ('260.0.0.0', False), + ('300.0.0.0', False), + ('0.0.0', False), + ('0.0.0.0.0', False), + ('example.com', False), + ]: + with self.subTest(host=host): + match = ancestor_hosts._IP_V4_REGEXP.match(host) + self.assertEqual(match is not None, expected) + + +class TestAncestorHosts(unittest.TestCase): + def test_good(self): + for host, expected in [ + ('example.com', ['example.com']), + ('a.example.com', ['a.example.com', 'example.com']), + ('a.b.example.com', [ + 'a.b.example.com', 'b.example.com', 'example.com']), + ('0.0.0.0', ['0.0.0.0']), + ('[::1]', ['[::1]']), + ]: + with self.subTest(host=host): + uris = list( + ancestor_hosts.ancestor_hosts(host=host)) + self.assertEqual(uris, expected) diff --git a/ref-engine-discovery.md b/ref-engine-discovery.md new file mode 100644 index 0000000..9b9e951 --- /dev/null +++ b/ref-engine-discovery.md @@ -0,0 +1,134 @@ +# OCI Ref-engine Discovery + +This is version 0.1 of this specification. + +To faciliate communication between image publishers and consumers, this specification defines a [ref-engine discovery](#ref-engine-discovery) protocol which publishers MAY use to direct consumers towards [reference engines](#ref-engine). +Publishers who choose not to support this specification can safely ignore the remainder of this document. + +Having retrieved a set of reference engines (via this and other protocols), consumers can use those ref engines to recover a set of [Merkle roots](#merkle-root) potentially associated with a given image name. +Consumers who choose not to support this specification can safely ignore the remainder of this document. +Consumers who choose to support this specification MAY attempt to discover and use ref engines via other channels, and only fall back to this protocol if those ref engines do not return a satisfactory Merkle root. + +## Glossary + +### Ref-engine discovery + +A service that suggests possible [ref engines](#ref-engine). +This specification defines a ref-engine discovery protocol. + +### CAS engine + +A service that provides access to [content-addressable storage][cas]. + +### Merkle root + +The root node in a [Merkle tree][Merkle-tree]. +In the OCI ecosystem, Merkle links are made via [descriptors][descriptor]. +The Merkle root may be a descriptor ([media type][media-type] [`application/vnd.oci.descriptor.v1+json`][descriptor]), or it may have a different media type. +Merkle roots may suggest [CAS engines](#cas-engine), e.g. via a `casEngines` entry in their JSON, but that is out of scope for ref-engine discovery. + +### Ref engine + +A service that maps an image name to a set of potential [Merkle roots](#merkle-root). + +## `oci-host-ref-engines` well-known URI registration + +This specification registers the `oci-host-ref-engines` well-known URI in the Well-Known URI Registery as defined by [RFC 5785][rfc5785]. + +URI suffix: `oci-host-ref-engines` + +Change controller: The [Open Container Initiative][OCI] + +Specification document(s): This specification + +Related information: None + +## Images associated with a host's `oci-host-ref-engines` + +Publishers SHOULD populate the `oci-host-ref-engines` resource with ref engines which are capable of resolving image names that match the [`host-based-image-name` rule](host-based-image-names.md) with a `host` part that matching their [fully qualified domain name][rfc1594-s5.2] and its subdomains or deeper descendants. +For example, https://b.example.com/.well-known/oci-host-ref-engines SHOULD prefer ref engines capable of resolving image names with `host` parts matching `b.example.com`, `a.b.example.com`, etc. +Some publishers MAY provide discovery services for generic image names (for example, to provide a company policy for ref-engine suggestions). +Those publishers MAY provide those recommendations via a [ref-engines resource](#ref-engines-media-types) at a URI of their choosing, but they SHOULD NOT serve the generic resource from `oci-host-ref-engines` to avoid distracting consumers following the protocol discussed in the following paragraph. + +Consumers discovering ref-engine for an image name that matches the [`host-based-image-name` rule](host-based-image-names.md) SHOULD request the `oci-host-ref-engines` resource from the host matching the `host` part. +If retrieving that resource fails for any reason, consumers SHOULD walk the DNS ancestors of `host`. +For example, if the `host` extracted from the image name is `a.b.example.com` and the well-known URI failed for `a.b.example.com`, the client would fall back to `b.example.com` and, if that too failed, to `example.com`. + +## Ref-engines media types + +Servers supporting the [`oci-host-ref-engines` URI](#oci-host-ref-engines-well-known-uri-registration) MUST support requests for media type [`application/vnd.oci.ref-engines.v1+json`](#ref-engines-objects). +Servers MAY support other media types using HTTP content negotiation, as described in [RFC 7231 section 3.4][rfc7231-s3.4] (which is [also supported over HTTP/2][rfc7540-s8]). + +## Ref-engines objects + +This section defines the `application/vnd.oci.ref-engines.v1+json` [media type][media-type]. +Content of this type MUST be a JSON object, as defined in [RFC 7159 section 4][rfc7159-s4]. +The object MAY include a `refEngines` entry. +If set, the `refEngines` entry MUST be an [array][rfc7159-s5]. +Each entry in the `refEngines` array MUST be an [objects][rfc7159-s4] with at least a `protocol` entry specifying the [ref-engine protocol](ref-engine-protocols.md). +Consumers SHOULD ignore entries which declare unsupported `protocol` values. +The order of entries in the array is not significant. + +The ref-engine discovery service MAY also include `casEngines` entry if it wants to supplement suggestions made by the ref engines. +If set, the `refEngines` entry MUST be an [array][rfc7159-s5]. +Each entry in the `refEngines` array MUST be an [objects][rfc7159-s4] with at least a `protocol` entry specifying the [cas-engine protocol](cas-engine-protocols.md). +Consumers SHOULD ignore entries which declare unsupported `protocol` values. +The order of entries in the array is not significant. + +### Example 1 + +``` +$ curl -H 'Accept: application/vnd.oci.ref-engines.v1+json' https://a.b.example.com/.well-known/oci-host-ref-engines +{ + "refEngines": [ + { + "protocol": "oci-index-template-v1", + "uri": "https://{host}/ref/{name}" + }, + { + "protocol": "docker", + "uri": "https://index.docker.io/v2", + "authUri": "https://auth.docker.io/token", + "authService": "registry.docker.io", + } + ] +} +``` + +The [`oci-index-template-v1` protocol](index-template.md) is [registered](ref-engine-protocols.md). +The `docker` protocol is currently [unregistered](ref-engine-protocols.md), and is given as sketch of a possible extention protocol. + +### Example 2 + +``` +$ curl -H 'Accept: application/vnd.oci.ref-engines.v1+json' https://example.com/.well-known/oci-host-ref-engines +{ + "refEngines": [ + { + "protocol": "oci-index-template-v1", + "uri": "https://{host}/ref/{name}" + } + ], + "casEngines": [ + { + "protocol": "oci-cas-template-v1", + "uri": "https://a.example.com/cas/{algorithm}/{encoded:2}/{encoded}" + } + ], +} +``` + +The [`oci-index-template-v1` protocol](index-template.md) is [registered](ref-engine-protocols.md). +The [`oci-cas-template-v1` protocol](cas-template.md) is [registered](cas-engine-protocols.md). + +[CAS]: https://en.wikipedia.org/wiki/Content-addressable_storage +[descriptor]: https://github.com/opencontainers/image-spec/blob/v1.0.0/descriptor.md +[media-type]: https://tools.ietf.org/html/rfc6838 +[Merkle-tree]: https://en.wikipedia.org/wiki/Merkle_tree +[OCI]: https://www.opencontainers.org/ +[rfc1594-s5.2]: https://tools.ietf.org/html/rfc1594#section-5 +[rfc5785]: https://tools.ietf.org/html/rfc5785 +[rfc7159-s4]: https://tools.ietf.org/html/rfc7159#section-4 +[rfc7159-s5]: https://tools.ietf.org/html/rfc7159#section-5 +[rfc7231-s3.4]: https://tools.ietf.org/html/rfc7231#section-3.4 +[rfc7540-s8]: https://tools.ietf.org/html/rfc7540#section-8 diff --git a/ref-engine-protocols.md b/ref-engine-protocols.md new file mode 100644 index 0000000..ac2e6a1 --- /dev/null +++ b/ref-engine-protocols.md @@ -0,0 +1,12 @@ +# Ref-Engine Protocols + +There are many possible [ref-engine](ref-engine-discovery.md#ref-engine) protocols. +Having identifiers for the protocols facilitates [ref-engine discovery](ref-engine-discovery.md#ref-engine-discovery) by allowing discovery services to [describe the protocol for suggested ref engines](ref-engine-discovery.md#ref-engines-objects). +Consumers can then prefer ref engines which implement their favorite protocol and use the appropriate API to connect to them. + +This section registers known protocol identifiers and maps them to their specification. +Anyone may submit new ref-engine protocol identifiers for registration. + +| Protocol identifier | Specification | +|-------------------------|-------------------------------------------------------------| +| `oci-index-template-v1` | [OCI index template protocol, version 1](index-template.md) |