Add distribution spec project proposal #35
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| # Abstract | ||
|
|
||
| The Docker registry protocol has become the defacto standard across the container registry world ([https://github.com/docker/distribution/blob/master/docs/spec/api.md](https://github.com/docker/distribution/blob/master/docs/spec/api.md)). | ||
|
|
||
| In the OCI, having a solid, common distribution specification with conformance testing will ensure long lasting security and interoperability throughout the container ecosystem. | ||
|
|
||
|
Member
There was a problem hiding this comment. See above discussion please add: |
||
| ## Proposal | ||
|
Member
There was a problem hiding this comment. One thing I would like to clarify is whether we will actually be making improvements to the spec over time (as per how we did things in the other specifications) or whether we're just going to freeze on the current v2.0 and do nothing much afterwards -- since according to SemVer we'd need to work on v3.0 for that. Also, it was my understanding that the submission of Docker Distribution as a specification would be something like "any future distribution spec we agree upon will have support for Docker Distribution" -- so as to unblock the distribution problem without locking us into never making significant progress on Distribution. But maybe I misunderstood?
Contributor
There was a problem hiding this comment. @cyphar I don't think there is anything in this protocol or proposal preventing "significant progress". The specification has been out and in the open since its inception and we have yet to receive any sizable improvement proposals. Most of the perceived limitations of this protocol are limitations in the client implementation that is part of docker. There are some good changes that we can make in backwards compatible ways:
In addition, to this, there is a lot more that can be integrated on top of this protocol as part of client resolution. Most of the issues around naming, signing and mirroring can be taken care of with no API changes. However, I think it should be clear that the initial specification activity should be about ensuring that the specification matches the current state of the world without breaking compatibility.
Contributor
There was a problem hiding this comment. I think what would be good to see is some clear instructions for the non-expert in terms of writing and working on specifications. Coming from this kind of boat, I have the general sentiment that I want to contribute, and I want to learn about the process to improve expertise in this sort of contribution. I would look for something akin to a CONTRIBUTING.md, or even a simple bullet list in a README that goes through logical steps. Like:
I suppose this is a technical standard for reviewing technical standards, haha. For example, a nice parallel is to look at something like JOSS that has complete orchestration via a little robot integration. The workflow is clearly defined for the reviewer and reviewee, and the assigned editor. The parties involved fill in the gaps in terms of opening and resolving issues, but there is always clear definition. I'm not sure how something like this might fit for our group, but it would be great to think about to get greater community involvement and discussion.
Contributor
Contributor
There was a problem hiding this comment. Thank @wking ! The general roles for a contributor and maintainer are well defined, but I'm wondering about a more specific "hand holding" how to contribute sort of document, even just a bullet list. If it's something that can be observed then I can observe, learn, and write something up. I'm mostly thinking about this to help new contributors such as myself to get into the ropes of the group :)
Contributor
There was a problem hiding this comment. Sure thing! A specific question before closing the discussion here. For this PR, given that we don't have code (nothing to test or critique), and it's a general text draft, what are the criteria we are using for evaluating it, and how do we know when it's ready to go? I think this is likely something that would be obvious for someone working on a lot of these drafts over time, and I'll just catch on, and it would be helpful if someone could jot down a few notes about these questions.
Contributor
There was a problem hiding this comment.
Ah, this repo. I thought you meant the coming distribution-spec repo. This repo could use a
Member
There was a problem hiding this comment. Right. Can you expand on these points (we had a short chat a while ago about them, but I don't think I fully understood what the plan was):
Also, I would still like to have some sort of Just on this point:
The reason for this may be unrelated to whether people want to make changes, or have worthwhile improvements. As you mentioned, a lot of the percieved issues with Distribution are actually because the Docker client doesn't expose those features -- and so people who want to improve distribution may not want to go through improving all layers of the Docker stack to do so. I guess what I'm saying is "let's not close the door on any improvement discussions", especially since this is now going to be an OCI spec and not a Docker one anymore.
Member
There was a problem hiding this comment. @cyphar what sort of language are you looking for to ensure the door is not closed for improvement discussions? How about after the interoperability statement adding:
Member
There was a problem hiding this comment.
To add to what @cyphar stated, I think that the pivot from a Docker spec to an OCI spec changes both the set and focus of the contributors; since the spec is no longer being driven by the Docker product's needs it can attract use-cases that would not have made sense for the Docker product. @mikebrow That language looks fairly reasonable to me. |
||
|
|
||
| TL;DR; Move [https://github.com/docker/distribution/tree/master/docs/spec](https://github.com/docker/distribution/tree/master/docs/spec) to [https://github.com/opencontainers/distribution-spec](https://github.com/opencontainers/distribution-spec) | ||
|
|
||
| This proposal covers the distribution API spec, and while it does not cover the code for the docker-registry, that implementation is considered the reference implementation. There are other implementations of this protocol, not all are open-source though (Google gcr.io, Amazon ECR, CoreOS Quay, Gitlab registry, JFrog Artifactory registry, Huawei Dockyard, etc). | ||
|
There was a problem hiding this comment. Is authentication in the scope of the image distribution? There was a problem hiding this comment. Also, shall we call out image signing (trusted image)?
Contributor
There was a problem hiding this comment. Image signing would be very useful for Singularity containers.
Member
There was a problem hiding this comment. @yuwaMSFT2 Unless I'm reading the current spec wrong, authentication is currently included.
Contributor
There was a problem hiding this comment.
The authentication for the registry protocol is just http-based authentication. There is a token authentication specification, which may be in scope, but there a lot of ways to actually implement it that may be context-specific.
This is not in scope. All resources are content-addressable and can be signed in external systems. Early versions of the specification and implementation had integrated signing, but there were a lot of problems with it. In practice, the enforcement needs to be in the hands of the client. This doesn't mean the registry couldn't be used to store signature blobs, but that would require more thought. In practice, we've found the best approach is to decouple this.
Contributor
There was a problem hiding this comment. For the token authentication - the fact that it is challenging suggests it's even more important to figure out. I would guess that more rather than fewer would want to use token auth, and perhaps the specification can have a default and then fall back cases for a client to implement that are guaranteed to work for most. Without that, we are in a situation of needing a custom implementation for each one, and that defeats the purpose of having a standard. Maybe we could try, and see how far we get? We can always fall back to sticking with just the basic. And agreed about the signing, given that it's likely very different! It would be good to come back to this at some future point (and as it's more commonly done and discussed).
Contributor
There was a problem hiding this comment.
I'm in favor of moving over enough of the auth spec to allow clients to authenticate with Docker's
Contributor
There was a problem hiding this comment. The issue with this approach is that each auth provider will then have to issue docker-style tokens to interact with the registry. With a more open approach, each registry can choose which providers are integrated. There is also the issue of the access control model: the model used in docker tokens is specific to the way docker implements the registry. Other providers may want to choose to have different token models. From the perspective of the client, most of this is opaque. They pull an image and authorize the pull through whatever flow makes sense for the context and provide an opaque value for an http header. |
||
|
|
||
| In the past when the topic of having an OCI specification around the distribution of container images was discussed, it was deferred as "let’s get the image format defined, meanwhile the industry will settle on a distribution standard". Fast forward, OCI image format is out and adopted, and the Registry v2 is the defacto standard. There is and will be use-cases for alternate methods and the future will likely hold creative ways to push, fetch and share container images, but right now this promotion serves to acknowledge by the OCI the current industry standard of distributing container images. | ||
|
|
||
| There is polish that is needed e.g. broken links to storage-driver docs, as well as making sections more generic regarding the OCI descriptors and media-types, but on the whole this is a lateral move. | ||
|
|
||
| ## Initial Maintainers | ||
|
|
||
| * Stephen Day <[email protected]> (@stevvooe) | ||
| * Vincent Batts <[email protected]> (@vbatts) | ||
| * Derek McGowan <[email protected]> (@dmcgowan) | ||
|
Contributor
There was a problem hiding this comment. With this proposal referencing project-template for the project's initial rules, we'll need a way to deal with project-template's Chief Maintainer role. Possibilities include:
My recommendation would be to remove the role (slightly more discussion here and here), but that PR has been dangling for 1.4 years, so I don't know how likely it is to land in the next month. Perhaps parallel work can land the Chief Maintainer removal, or opencontainers/project-template#41 may end up with the TOB in charge of project-template. With project-template (currently) outside of direct TOB control, the simplest approach would be to appoint a Chief Maintainer here, as that entirely within the TOB's control.
Contributor
There was a problem hiding this comment.
This happened, so nothing left to do for this specific point. |
||
|
|
||
| Additional Maintainers to consider: | ||
|
Contributor
There was a problem hiding this comment. The TOB needs to pick an initial maintainer set. I think these suggestions are for the TOB, and should be reviewed and either added to the initial maintainer list or dropped before the vote. A three-person initial maintainer set can add other maintainers on their own, so the TOB can drop tge whole consideration list and punt to the initial maintainers if it wants.
Member
There was a problem hiding this comment. Same. How are TOB members supposed to interpret this list? Maintainers are picked based on current and past contributions to a codebase or area of expertise and usually their maintainer vote comes with justification of why they should be a maintainer in a project.
Contributor
There was a problem hiding this comment. @caniszczyk, were we just dropping this list, and leaving it up to the above-three maintainers to add additional maintainers? |
||
|
|
||
| * Ahmet Alp Balkan (Google) | ||
| * Matt Moore (Google) | ||
| * Yuwa (MSFT) | ||
| * Clayton Coleman (Red Hat) | ||
| * Antonio Murdaca (@runcom) (Red Hat) | ||
| * Samuel Karp (@samuelkarp) (AWS) | ||
| * Mike Brown (IBM) | ||
| * Jimmy Zelinskie [email protected] (@jzelinskie) | ||
| * Liu Genping <[[email protected]](mailto:[email protected])> | ||
| * Vanessa Sochat (@vsoch) (Stanford) <[email protected]> | ||
| * Eduardo Arango (@ArangoGutierrez) (Sylabs) <[email protected]> | ||
|
|
||
|
Contributor
There was a problem hiding this comment. For completeness, choose any/all that is needed:
|
||
| ## Code of Conduct | ||
|
|
||
| This project would incorporate (by reference) the OCI Code of Conduct ([https://github.com/opencontainers/tob/blob/master/code-of-conduct.md](https://github.com/opencontainers/tob/blob/master/code-of-conduct.md)). | ||
|
|
||
| ## Governance and Releases | ||
|
|
||
| This project would incorporate the Governance and Releases processes from the OCI project template: [https://github.com/opencontainers/project-template](https://github.com/opencontainers/project-template). | ||
|
|
||
| ## Project Communications | ||
|
|
||
| Both of the proposed projects would continue to use existing channels in use by the OCI developer community for communication including: | ||
|
|
||
| * GitHub for issues and pull requests | ||
| * The [email protected] email list | ||
| * The monthly OCI developer community conference call | ||
|
Contributor
There was a problem hiding this comment. Does starting at v2 (as suggested below) get us out of the pre-1.0 weekly meeting recommendation? I'm not sure if the "we're starting at v2" approach was considered when writing those docs (I certainly hadn't considered it).
Member
There was a problem hiding this comment. It was discussed in the google doc
Contributor
There was a problem hiding this comment. hey @wking here you go! If you follow the link at the top, click on "Comments" in the top right, you can explore some of the discussion.
Contributor
There was a problem hiding this comment. I'm fine with starting at v2. I'm just not clear on whether monthly meetings are the right target level for a new spec (regardless of the number we put on our initial release). The discussion in this PR suggests folks are going to have lots of ideas. Project-template suggests weekly meetings during pre-1.0 development (presumably because the roadmap is less obvious/established then). And there's no reason that roadmap discussions and such couldn't happen asynchronously (but then maybe we want to drop the meeting recommendation from project-template?). Anyhow, I'd just like to make sure that, if this project starts out with monthly meetings, it was a conscious decision that we didn't expect to need weekly meetings.
Contributor
There was a problem hiding this comment.
The specification is not new. It was already discussed in the open and 100+ comments.
Member
There was a problem hiding this comment. @wking sounds like something to discuss on the next monthly meeting :-)
Member
There was a problem hiding this comment. While there is plenty of discussion here, not such an overhaul needed as to change increase from monthly. I am sure we'll adjust the meeting schedule as needed |
||
| * The #OpenContainers freenode IRC channel | ||
|
|
||
| ## Versioning / Roadmap | ||
|
|
||
| The API spec is currently considered v2 and we will start the specification at v2.0. Fewer places to change and compare, and it would keep with it being a lateral move. | ||
|
Member
There was a problem hiding this comment. I'm not sure we should start with How about starting with
Contributor
There was a problem hiding this comment.
Somewhat complicating this approach is that the fact that Docker's v2 API has been receiving lettered minor/patch bumps. Presumably those will all be considered part of the 2.0.x series? And if the goal is to start using SemVer (which sounds like a useful goal to me), we'll need to specify that and wait for some deprecation period before cutting v2.1, or we'll have all the 2.0 clients choke and die when the version-check endpoint sets
Contributor
There was a problem hiding this comment. @wking In general, since the subversions are backwards compatible, we do not expose the subversions. In general, we should move away from path-level versioning and favor type-level versioning.
Member
There was a problem hiding this comment. @AkihiroSuda are you saying for that header, or just tagged releases or both?
Member
There was a problem hiding this comment. FWIW, Docker v2-2 is the most recent protocol -- I hope everyone here knows that and has just been using 2.1 as an example. |
||
|
|
||
| ## Frequently Asked Questions (FAQ) | ||
|
|
||
| * Does this include the code of the docker-registry? | ||
| * No. This is an API specification discussion. | ||
|
|
||
| ## Related GitHub Issues | ||
|
There was a problem hiding this comment. What about the proposal of new APIs to the spec? Shall we do it after the vote as future incremental improvements, or shall we raise and discuss them now? From our experience of running our registry service (Azure Container Registry), there are quite some popular feedbacks/lessons we learned and would like to see if they could be included in the future registry spec:
We would like to learn if there is any opportunity that these feedback could be included in the distribution spec. Thanks!
Contributor
There was a problem hiding this comment. Image size and format would also be very useful, beyond sniffing with a
Member
There was a problem hiding this comment. @yuwaMSFT2 @vsoch interesting extensions. See above proposal to add a sentence about discussing and scheduling in extensions without going into specific detail. There was a problem hiding this comment. @vsoch no the metadata of the manifest is not what I meant for tenant.
Contributor
There was a problem hiding this comment. @mikebrow this looks good to me!
Contributor
There was a problem hiding this comment. @yuwaMSFT2 A lot of the "extensions" you've referenced are really about content management, rather than content distribution. This API has almost always been about content distribution. Adding these extensions to the specification provides little room for vendors to implement these kinds of functionality in a way that fits their platform.
@vsoch None of this is a part of this specification. Even the hub behavior @yuwaMSFT2 was referencing is the hubs implementation. This specification allows arbitrary paths. Other things, like Let's make sure this exercise is critiquing the specification and not Docker's implementation of it.
Contributor
There was a problem hiding this comment. @yuwaMSFT2 agreed about content management vs. distribution, I've never distinguished those two before. In that the uri is serving as an entry point into a registry, even if it doesn't conform to a specific namespace, tag, arguably I should be able to (knowing the API conforms to the registry) programatically predict the base uri to be calling based on this alone. It could be as general as a general flow of regular expressions to follow, or as specific as a discrete set of different kinds (e.g., abstract uri vs. an arbitrary path). It would be a stronger specification to have some level of predictibility here, otherwise I still need to write a custom thing (after checking!) for each registry endpoint to even interact with most endpoints.
Contributor
There was a problem hiding this comment.
The two systems have vastly different requirements. For example, even though the Hub looks like its the same system, they are logically separate between the registry and the hub ui, in practice. This prevents the registry deployment from getting complicated. All of the UI and management of images gets implemented in a separate system, allowing them to grow in features and functionality. I'm still not quite understanding what you're talking about regarding URIs: the format that you're talking about parsing has nothing to do with this specification. Different systems may implement completely different behaviors and formats, depending on their opinion.
Contributor
There was a problem hiding this comment. @vsoch @yuwaMSFT2 I expanded a bit on the divide between content distribution and management in #37 (comment). I'll repost here, as there may have been some confusion. This API had everything in it required to integrate with a container runtime (ie. pull an image, figure out what tags are in a repo, etc.) and APIs required to integrate with content management systems. This includes listing repositories and tags. There is also a notification API that enables in the registry implementation, but isn't a part of this specification (there may be an argument about pulling this in). |
||
|
|
||
| * Simplifies tag listing: docker/distribution#2169 | ||
|
Contributor
There was a problem hiding this comment. Let's add a statement here about what these PRs mean. Are these additions required by the proposal? They have been vetted and implemented, so I would think they are a good candidate. |
||
| * Allows listing of manifests: docker/distribution#2199 | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still think we want to pin this to a specific version. For example:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That should be an implementation detail of the proposal. The implementors of the proposal should choose the version to adopt so this can be changed without TOB involvement.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the problem is that the hyperlink will be broken when the spec is moved to the OCI distribution-spec repo.
I think keeping the hyperlink permanently after the migration is good for recording the history.
So +1 for pinning.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And as long as the TOB-selected version isn't far behind (how far will the spec move during a week of voting?), it should be easy for the new maintainets to catch up on any subsequent drift.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can use such as language when pointing to a particular version...