Skip to content

Commit c11d80c

Browse files
authored
Merge pull request #16 from akashsinghal/addReferrerGC
Add Referrers Garbage Collection
2 parents f98a060 + 8e19c16 commit c11d80c

22 files changed

+857
-130
lines changed

.github/workflows/oras-release.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ on:
44
push:
55
tags:
66
- "v[0-9]+.[0-9]+.[0-9]+-alpha"
7-
- "v[0-9]+.[0-9]+.[0-9]+-rc"
7+
- "v[0-9]+.[0-9]+.[0-9]+-rc.[0-9]+"
88

99
jobs:
1010
publish:

docs/extensions.md

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
---
2+
description: High level discussion of extensions
3+
keywords: registry, extension, handlers, repository, distribution, artifacts
4+
title: Extensions
5+
---
6+
7+
This document serves as a high level discussion of the implementation of the extensions framework defined in the [OCI Distribution spec](https://github.com/opencontainers/distribution-spec/tree/main/extensions).
8+
9+
## Extension Interface
10+
11+
The `Extension` interface is introduced in the new `extension` package. It defines methods to access the extension's namespace-specific attributes such as the Name, Url defining the extension namespace, and the Description of the namespace. It defines route enumeration at the Registry and Repository level. It also encases the `ExtendedStorage` interface which defines the methods requires to extend the underlying storage functionality of the registry.
12+
13+
```
14+
type Extension interface {
15+
storage.ExtendedStorage
16+
// GetRepositoryRoutes returns a list of extension routes scoped at a repository level
17+
GetRepositoryRoutes() []ExtensionRoute
18+
// GetRegistryRoutes returns a list of extension routes scoped at a registry level
19+
GetRegistryRoutes() []ExtensionRoute
20+
// GetNamespaceName returns the name associated with the namespace
21+
GetNamespaceName() string
22+
// GetNamespaceUrl returns the url link to the documentation where the namespace's extension and endpoints are defined
23+
GetNamespaceUrl() string
24+
// GetNamespaceDescription returns the description associated with the namespace
25+
GetNamespaceDescription() string
26+
}
27+
```
28+
29+
The `ExtendedStorage` interface defines methods that specify storage-specific handlers. Each extension will implement a handler extending the functionality. The interface can be expanded in the future to consider new handler types.
30+
`GetManifestHandlers` is used to return new `ManifestHandlers` defined by each of the extensions.
31+
`GetGarbageCollectionHandlers` is used to return `GCExtensionHandler` implemented by each extension.
32+
33+
```
34+
type ExtendedStorage interface {
35+
// GetManifestHandlers returns the list of manifest handlers that handle custom manifest formats supported by the extension
36+
GetManifestHandlers(
37+
repo Repository,
38+
blobStore BlobStore) []ManifestHandler
39+
// GetGarbageCollectHandlers returns the GCExtensionHandlers that handles custom garbage collection behavior for the extension.
40+
GetGarbageCollectionHandlers() []GCExtensionHandler
41+
}
42+
```
43+
44+
The `GCExtensionHandler` interface defines three methods that are used in the garbage colection mark and sweep process. The `Mark` method is invoked for each `GCExtensionHandler` after the existing mark process finishes in `MarkAndSweep`. It is used to determine if the manifest and blobs should have their temporary ref count incremented in the case of an artifact manifest, or if the manifest and it's referrers should be recursively indexed for deletion in the case of a non-artifact manifest. `OnManifestDelete` is invoked to extend the `RemoveManifest` functionality for the `Vacuum`. New or special-cased manifests may require custom manifest deletion which can be defined with this method. `SweepBlobs` is used to add artifact manifest/blobs to the original `markSet`. These blobs are retained after determining their ref count is still positive.
45+
46+
```
47+
type GCExtensionHandler interface {
48+
Mark(ctx context.Context,
49+
repository distribution.Repository,
50+
storageDriver driver.StorageDriver,
51+
registry distribution.Namespace,
52+
manifest distribution.Manifest,
53+
manifestDigest digest.Digest,
54+
dryRun bool,
55+
removeUntagged bool) (bool, error)
56+
OnManifestDelete(ctx context.Context,
57+
storageDriver driver.StorageDriver,
58+
registry distribution.Namespace,
59+
dgst digest.Digest,
60+
repositoryName string) error
61+
SweepBlobs(ctx context.Context,
62+
markSet map[digest.Digest]struct{}) map[digest.Digest]struct{}
63+
}
64+
```
65+
66+
## Registering Extensions
67+
68+
Extensions are defined in the configuration yaml.
69+
70+
### Sample Extension Configuration YAML
71+
```
72+
# Configuration for extensions. It follows the below schema
73+
# extensions
74+
# namespace:
75+
# configuration for the extension and its components in any schema specific to that namespace
76+
extensions:
77+
oci:
78+
ext:
79+
- discover # enable the discovery extension
80+
```
81+
82+
Each `Extension` defined must call the `RegisterExtension` method to register an extension initialization function with the extension namespace name. The registered extension list is then used during configuration parsing to get and initialize the specified extension. (`GetExtension`)
83+
84+
```
85+
// InitExtension is the initialize function for creating the extension namespace
86+
type InitExtension func(ctx context.Context, storageDriver driver.StorageDriver, options configuration.ExtensionConfig) (Extension, error)
87+
88+
// RegisterExtension is used to register an InitExtension for
89+
// an extension with the given name.
90+
func RegisterExtension(name string, initFunc InitExtension)
91+
92+
// GetExtension constructs an extension with the given options using the given name.
93+
func GetExtension(ctx context.Context, name string, storageDriver driver.StorageDriver, options configuration.ExtensionConfig) (Extension, error)
94+
```
95+
96+
Each `Extension` defines an `ExtensionRoute` which contains the new `<namespace>/<extension>/<component>` route attributes. Furthermore, the route `Descriptor` and `Dispatcher` are used to register the new route to the application.
97+
98+
```
99+
type ExtensionRoute struct {
100+
// Namespace is the name of the extension namespace
101+
Namespace string
102+
// Extension is the name of the extension under the namespace
103+
Extension string
104+
// Component is the name of the component under the extension
105+
Component string
106+
// Descriptor is the route descriptor that gives its path
107+
Descriptor v2.RouteDescriptor
108+
// Dispatcher if present signifies that the route is http route with a dispatcher
109+
Dispatcher RouteDispatchFunc
110+
}
111+
112+
// RouteDispatchFunc is the http route dispatcher used by the extension route handlers
113+
type RouteDispatchFunc func(extContext *ExtensionContext, r *http.Request) http.Handler
114+
```
115+
116+
117+

docs/garbage-collection.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,4 +121,4 @@ blob eligible for deletion: sha256:7e15ce58ccb2181a8fced7709e9893206f0937cc9543b
121121
blob eligible for deletion: sha256:87192bdbe00f8f2a62527f36bb4c7c7f4eaf9307e4b87e8334fb6abec1765bcb
122122
blob eligible for deletion: sha256:b549a9959a664038fc35c155a95742cf12297672ca0ae35735ec027d55bf4e97
123123
blob eligible for deletion: sha256:f251d679a7c61455f06d793e43c06786d7766c88b8c24edf242b2c08e3c3f599
124-
```
124+
```

registry/extension/distribution/manifests.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ type manifestsGetAPIResponse struct {
1919

2020
// manifestHandler handles requests for manifests under a manifest name.
2121
type manifestHandler struct {
22-
*extension.Context
22+
*extension.ExtensionContext
2323
storageDriver driver.StorageDriver
2424
}
2525

registry/extension/distribution/registry.go

Lines changed: 18 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ type distributionOptions struct {
3434
}
3535

3636
// newDistNamespace creates a new extension namespace with the name "distribution"
37-
func newDistNamespace(ctx context.Context, storageDriver driver.StorageDriver, options configuration.ExtensionConfig) (extension.Namespace, error) {
37+
func newDistNamespace(ctx context.Context, storageDriver driver.StorageDriver, options configuration.ExtensionConfig) (extension.Extension, error) {
3838

3939
optionsYaml, err := yaml.Marshal(options)
4040
if err != nil {
@@ -67,7 +67,7 @@ func newDistNamespace(ctx context.Context, storageDriver driver.StorageDriver, o
6767

6868
func init() {
6969
// register the extension namespace.
70-
extension.Register(namespaceName, newDistNamespace)
70+
extension.RegisterExtension(namespaceName, newDistNamespace)
7171
}
7272

7373
// GetManifestHandlers returns a list of manifest handlers that will be registered in the manifest store.
@@ -76,12 +76,17 @@ func (o *distributionNamespace) GetManifestHandlers(repo distribution.Repository
7676
return []storage.ManifestHandler{}
7777
}
7878

79+
func (o *distributionNamespace) GetGarbageCollectionHandlers() []storage.GCExtensionHandler {
80+
// This extension doesn't extend any garbage collection operations.
81+
return []storage.GCExtensionHandler{}
82+
}
83+
7984
// GetRepositoryRoutes returns a list of extension routes scoped at a repository level
80-
func (d *distributionNamespace) GetRepositoryRoutes() []extension.Route {
81-
var routes []extension.Route
85+
func (d *distributionNamespace) GetRepositoryRoutes() []extension.ExtensionRoute {
86+
var routes []extension.ExtensionRoute
8287

8388
if d.manifestsEnabled {
84-
routes = append(routes, extension.Route{
89+
routes = append(routes, extension.ExtensionRoute{
8590
Namespace: namespaceName,
8691
Extension: extensionName,
8792
Component: manifestsComponentName,
@@ -100,7 +105,7 @@ func (d *distributionNamespace) GetRepositoryRoutes() []extension.Route {
100105
}
101106

102107
if d.tagHistoryEnabled {
103-
routes = append(routes, extension.Route{
108+
routes = append(routes, extension.ExtensionRoute{
104109
Namespace: namespaceName,
105110
Extension: extensionName,
106111
Component: tagHistoryComponentName,
@@ -134,7 +139,7 @@ func (d *distributionNamespace) GetRepositoryRoutes() []extension.Route {
134139

135140
// GetRegistryRoutes returns a list of extension routes scoped at a registry level
136141
// There are no registry scoped routes exposed by this namespace
137-
func (d *distributionNamespace) GetRegistryRoutes() []extension.Route {
142+
func (d *distributionNamespace) GetRegistryRoutes() []extension.ExtensionRoute {
138143
return nil
139144
}
140145

@@ -153,21 +158,21 @@ func (d *distributionNamespace) GetNamespaceDescription() string {
153158
return namespaceDescription
154159
}
155160

156-
func (d *distributionNamespace) tagHistoryDispatcher(ctx *extension.Context, r *http.Request) http.Handler {
161+
func (d *distributionNamespace) tagHistoryDispatcher(ctx *extension.ExtensionContext, r *http.Request) http.Handler {
157162
tagHistoryHandler := &tagHistoryHandler{
158-
Context: ctx,
159-
storageDriver: d.storageDriver,
163+
ExtensionContext: ctx,
164+
storageDriver: d.storageDriver,
160165
}
161166

162167
return handlers.MethodHandler{
163168
"GET": http.HandlerFunc(tagHistoryHandler.getTagManifestDigests),
164169
}
165170
}
166171

167-
func (d *distributionNamespace) manifestsDispatcher(ctx *extension.Context, r *http.Request) http.Handler {
172+
func (d *distributionNamespace) manifestsDispatcher(ctx *extension.ExtensionContext, r *http.Request) http.Handler {
168173
manifestsHandler := &manifestHandler{
169-
Context: ctx,
170-
storageDriver: d.storageDriver,
174+
ExtensionContext: ctx,
175+
storageDriver: d.storageDriver,
171176
}
172177

173178
return handlers.MethodHandler{

registry/extension/distribution/taghistory.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ type tagHistoryAPIResponse struct {
2020

2121
// manifestHandler handles requests for manifests under a manifest name.
2222
type tagHistoryHandler struct {
23-
*extension.Context
23+
*extension.ExtensionContext
2424
storageDriver driver.StorageDriver
2525
}
2626

registry/extension/extension.go

Lines changed: 24 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
package extension
22

33
import (
4-
c "context"
4+
"context"
55
"fmt"
66
"net/http"
77

@@ -13,9 +13,9 @@ import (
1313
"github.com/distribution/distribution/v3/registry/storage/driver"
1414
)
1515

16-
// Context contains the request specific context for use in across handlers.
17-
type Context struct {
18-
c.Context
16+
// ExtensionContext contains the request specific context for use in across handlers.
17+
type ExtensionContext struct {
18+
context.Context
1919

2020
// Registry is the base namespace that is used by all extension namespaces
2121
Registry distribution.Namespace
@@ -26,10 +26,10 @@ type Context struct {
2626
}
2727

2828
// RouteDispatchFunc is the http route dispatcher used by the extension route handlers
29-
type RouteDispatchFunc func(extContext *Context, r *http.Request) http.Handler
29+
type RouteDispatchFunc func(extContext *ExtensionContext, r *http.Request) http.Handler
3030

31-
// Route describes an extension route.
32-
type Route struct {
31+
// ExtensionRoute describes an extension route.
32+
type ExtensionRoute struct {
3333
// Namespace is the name of the extension namespace
3434
Namespace string
3535
// Extension is the name of the extension under the namespace
@@ -42,13 +42,14 @@ type Route struct {
4242
Dispatcher RouteDispatchFunc
4343
}
4444

45-
// Namespace is the namespace that is used to define extensions to the distribution.
46-
type Namespace interface {
45+
// Extension is the interface that is used to define extensions to the distribution.
46+
type Extension interface {
4747
storage.ExtendedStorage
48+
// ExtensionService
4849
// GetRepositoryRoutes returns a list of extension routes scoped at a repository level
49-
GetRepositoryRoutes() []Route
50+
GetRepositoryRoutes() []ExtensionRoute
5051
// GetRegistryRoutes returns a list of extension routes scoped at a registry level
51-
GetRegistryRoutes() []Route
52+
GetRegistryRoutes() []ExtensionRoute
5253
// GetNamespaceName returns the name associated with the namespace
5354
GetNamespaceName() string
5455
// GetNamespaceUrl returns the url link to the documentation where the namespace's extension and endpoints are defined
@@ -57,8 +58,8 @@ type Namespace interface {
5758
GetNamespaceDescription() string
5859
}
5960

60-
// InitExtensionNamespace is the initialize function for creating the extension namespace
61-
type InitExtensionNamespace func(ctx c.Context, storageDriver driver.StorageDriver, options configuration.ExtensionConfig) (Namespace, error)
61+
// InitExtension is the initialize function for creating the extension namespace
62+
type InitExtension func(ctx context.Context, storageDriver driver.StorageDriver, options configuration.ExtensionConfig) (Extension, error)
6263

6364
// EnumerateExtension specifies extension information at the namespace level
6465
type EnumerateExtension struct {
@@ -68,10 +69,10 @@ type EnumerateExtension struct {
6869
Endpoints []string `json:"endpoints"`
6970
}
7071

71-
var extensions map[string]InitExtensionNamespace
72-
var extensionsNamespaces map[string]Namespace
72+
var extensions map[string]InitExtension
73+
var extensionsNamespaces map[string]Extension
7374

74-
func EnumerateRegistered(ctx Context) (enumeratedExtensions []EnumerateExtension) {
75+
func EnumerateRegistered(ctx ExtensionContext) (enumeratedExtensions []EnumerateExtension) {
7576
for _, namespace := range extensionsNamespaces {
7677
enumerateExtension := EnumerateExtension{
7778
Name: fmt.Sprintf("_%s", namespace.GetNamespaceName()),
@@ -102,11 +103,11 @@ func EnumerateRegistered(ctx Context) (enumeratedExtensions []EnumerateExtension
102103
return enumeratedExtensions
103104
}
104105

105-
// Register is used to register an InitExtensionNamespace for
106-
// an extension namespace with the given name.
107-
func Register(name string, initFunc InitExtensionNamespace) {
106+
// RegisterExtension is used to register an InitExtension for
107+
// an extension with the given name.
108+
func RegisterExtension(name string, initFunc InitExtension) {
108109
if extensions == nil {
109-
extensions = make(map[string]InitExtensionNamespace)
110+
extensions = make(map[string]InitExtension)
110111
}
111112

112113
if _, exists := extensions[name]; exists {
@@ -116,11 +117,11 @@ func Register(name string, initFunc InitExtensionNamespace) {
116117
extensions[name] = initFunc
117118
}
118119

119-
// Get constructs an extension namespace with the given options using the given name.
120-
func Get(ctx c.Context, name string, storageDriver driver.StorageDriver, options configuration.ExtensionConfig) (Namespace, error) {
120+
// GetExtension constructs an extension with the given options using the given name.
121+
func GetExtension(ctx context.Context, name string, storageDriver driver.StorageDriver, options configuration.ExtensionConfig) (Extension, error) {
121122
if extensions != nil {
122123
if extensionsNamespaces == nil {
123-
extensionsNamespaces = make(map[string]Namespace)
124+
extensionsNamespaces = make(map[string]Extension)
124125
}
125126

126127
if initFunc, exists := extensions[name]; exists {

registry/extension/oci/discover.go

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ type discoverGetAPIResponse struct {
1616

1717
// extensionHandler handles requests for manifests under a manifest name.
1818
type extensionHandler struct {
19-
*extension.Context
19+
*extension.ExtensionContext
2020
storageDriver driver.StorageDriver
2121
}
2222

@@ -26,7 +26,7 @@ func (eh *extensionHandler) getExtensions(w http.ResponseWriter, r *http.Request
2626
w.Header().Set("Content-Type", "application/json")
2727

2828
// get list of extension information seperated at the namespace level
29-
enumeratedExtensions := extension.EnumerateRegistered(*eh.Context)
29+
enumeratedExtensions := extension.EnumerateRegistered(*eh.ExtensionContext)
3030

3131
// remove the oci extension so it's not returned by discover
3232
ociExtensionName := fmt.Sprintf("_%s", namespaceName)

0 commit comments

Comments
 (0)