remove support for non-distributable artifacts and deprecate API fields and config

Non-distributable artifacts (foreign layers) were introduced in commit
05bd043 to accommodate Windows images,
for which the EULA did not allow layers to be distributed through registries
other than those hosted by Microsoft. The concept of foreign / non-distributable
layers was adopted by the OCI distribution spec in [oci#233].

These restrictions were relaxed later to allow distributing these images
through non-public registries, for which a configuration was added in the
daemon in 67fdf57. In 2022, Microsoft updated
the EULA and [removed these restrictions altogether][1], and the OCI distribution
spec deprecated the  functionality in [oci#965].

In 2023, Microsoft [removed the use of foreign data layers][2] for their images,
making this functionality obsolete.

This patch:

- Deprecates the `--allow-nondistributable-artifacts` daemon flag and corresponding
  `allow-nondistributable-artifacts` field in `daemon.json`. Setting either
  option will no longer take an effect, but a deprecation warning log is added
  to raise awareness about the deprecation. This warning is planned to become
  an error in the next release.
- Deprecates the `RegistryConfig.AllowNondistributableArtifactsCIDRs` and
  `RegistryConfig.AllowNondistributableArtifactsHostnames` fields in the
  `GET /info` API response. For API version v1.48 and lower, the fields are
  still included in the response, but always `null`. In API version v1.49 and
  higher, the field will be omitted entirely.
- Deprecates the `api/types/registry/ServiceConfig.AllowNondistributableArtifactsCIDRs`
  field.
- Deprecates the `api/types/registry/ServiceConfig.AllowNondistributableArtifactsHostnames`
  field.
- Deprecates the `registry.ServiceOptions.AllowNondistributableArtifacts` field.

[oci#233]: opencontainers/image-spec#233
[oci#965]: opencontainers/image-spec#965
[1]: https://techcommunity.microsoft.com/blog/containers/announcing-windows-container-base-image-redistribution-rights-change/3645201
[2]: https://techcommunity.microsoft.com/blog/containers/announcing-removal-of-foreign-layers-from-windows-container-images/3846833

Signed-off-by: Sebastiaan van Stijn <[email protected]>

Author: thaJeztah

api/server/router/system/system_routes.go

@@ -100,6 +100,12 @@ func (s *systemRouter) getInfo(ctx context.Context, w http.ResponseWriter, r *ht
 			// Containerd field introduced in API v1.46.
 			info.Containerd = nil
 		}
+		if versions.LessThan(version, "1.47") {
+			// Field is omitted in API 1.48 and up, but should still be included
+			// in older versions, even if no values are set.
+			info.RegistryConfig.AllowNondistributableArtifactsCIDRs = []*registry.NetIPNet{}
+			info.RegistryConfig.AllowNondistributableArtifactsHostnames = []string{}
+		}
 
 		// TODO(thaJeztah): Expected commits are deprecated, and should no longer be set in API 1.49.
 		info.ContainerdCommit.Expected = info.ContainerdCommit.ID //nolint:staticcheck // ignore SA1019: field is deprecated, but still used on API < v1.49.

api/swagger.yaml

@@ -5983,55 +5983,27 @@ definitions:
           List of IP ranges to which nondistributable artifacts can be pushed,
           using the CIDR syntax [RFC 4632](https://tools.ietf.org/html/4632).
 
-          Some images (for example, Windows base images) contain artifacts
-          whose distribution is restricted by license. When these images are
-          pushed to a registry, restricted artifacts are not included.
-
-          This configuration override this behavior, and enables the daemon to
-          push nondistributable artifacts to all registries whose resolved IP
-          address is within the subnet described by the CIDR syntax.
-
-          This option is useful when pushing images containing
-          nondistributable artifacts to a registry on an air-gapped network so
-          hosts on that network can pull the images without connecting to
-          another server.
-
-          > **Warning**: Nondistributable artifacts typically have restrictions
-          > on how and where they can be distributed and shared. Only use this
-          > feature to push artifacts to private registries and ensure that you
-          > are in compliance with any terms that cover redistributing
-          > nondistributable artifacts.
+          <p><br /></p>
 
+          > **Deprecated**: Pushing nondistributable artifacts is now always enabled
+          > and this field is always `null`. This field will be removed in a API v1.49.
         type: "array"
         items:
           type: "string"
-        example: ["::1/128", "127.0.0.0/8"]
+        example: []
       AllowNondistributableArtifactsHostnames:
         description: |
           List of registry hostnames to which nondistributable artifacts can be
           pushed, using the format `<hostname>[:<port>]` or `<IP address>[:<port>]`.
 
-          Some images (for example, Windows base images) contain artifacts
-          whose distribution is restricted by license. When these images are
-          pushed to a registry, restricted artifacts are not included.
-
-          This configuration override this behavior for the specified
-          registries.
-
-          This option is useful when pushing images containing
-          nondistributable artifacts to a registry on an air-gapped network so
-          hosts on that network can pull the images without connecting to
-          another server.
+          <p><br /></p>
 
-          > **Warning**: Nondistributable artifacts typically have restrictions
-          > on how and where they can be distributed and shared. Only use this
-          > feature to push artifacts to private registries and ensure that you
-          > are in compliance with any terms that cover redistributing
-          > nondistributable artifacts.
+          > **Deprecated**: Pushing nondistributable artifacts is now always enabled
+          > and this field is always `null`. This field will be removed in a API v1.49.
         type: "array"
         items:
           type: "string"
-        example: ["registry.internal.corp.example.com:3000", "[2001:db8:a0b:12f0::1]:443"]
+        example: []
       InsecureRegistryCIDRs:
         description: |
           List of IP ranges of insecure registries, using the CIDR syntax

api/types/registry/registry.go

@@ -9,11 +9,29 @@ import (
 
 // ServiceConfig stores daemon registry services configuration.
 type ServiceConfig struct {
-	AllowNondistributableArtifactsCIDRs     []*NetIPNet
-	AllowNondistributableArtifactsHostnames []string
-	InsecureRegistryCIDRs                   []*NetIPNet           `json:"InsecureRegistryCIDRs"`
-	IndexConfigs                            map[string]*IndexInfo `json:"IndexConfigs"`
-	Mirrors                                 []string
+	AllowNondistributableArtifactsCIDRs     []*NetIPNet `json:"AllowNondistributableArtifactsCIDRs,omitempty"`     // Deprecated: non-distributable artifacts are deprecated and enabled by default. This field will be removed in the next release.
+	AllowNondistributableArtifactsHostnames []string    `json:"AllowNondistributableArtifactsHostnames,omitempty"` // Deprecated: non-distributable artifacts are deprecated and enabled by default. This field will be removed in the next release.
+
+	InsecureRegistryCIDRs []*NetIPNet           `json:"InsecureRegistryCIDRs"`
+	IndexConfigs          map[string]*IndexInfo `json:"IndexConfigs"`
+	Mirrors               []string
+}
+
+// MarshalJSON implements a custom marshaler to include legacy fields
+// in API responses.
+func (sc ServiceConfig) MarshalJSON() ([]byte, error) {
+	tmp := map[string]interface{}{
+		"InsecureRegistryCIDRs": sc.InsecureRegistryCIDRs,
+		"IndexConfigs":          sc.IndexConfigs,
+		"Mirrors":               sc.Mirrors,
+	}
+	if sc.AllowNondistributableArtifactsCIDRs != nil {
+		tmp["AllowNondistributableArtifactsCIDRs"] = nil
+	}
+	if sc.AllowNondistributableArtifactsHostnames != nil {
+		tmp["AllowNondistributableArtifactsHostnames"] = nil
+	}
+	return json.Marshal(tmp)
 }
 
 // NetIPNet is the net.IPNet type, which can be marshalled and

api/types/registry/registry_test.go

@@ -0,0 +1,30 @@
+package registry
+
+import (
+	"encoding/json"
+	"testing"
+
+	"gotest.tools/v3/assert"
+	is "gotest.tools/v3/assert/cmp"
+)
+
+func TestServiceConfigMarshalLegacyFields(t *testing.T) {
+	t.Run("without legacy fields", func(t *testing.T) {
+		b, err := json.Marshal(&ServiceConfig{})
+		assert.NilError(t, err)
+		const expected = `{"IndexConfigs":null,"InsecureRegistryCIDRs":null,"Mirrors":null}`
+		assert.Check(t, is.Equal(string(b), expected), "Legacy nondistributable-artifacts fields should be omitted in output")
+	})
+
+	// Legacy fields should be returned when set to an empty slice. This is
+	// used for API versions < 1.49.
+	t.Run("with legacy fields", func(t *testing.T) {
+		b, err := json.Marshal(&ServiceConfig{
+			AllowNondistributableArtifactsCIDRs:     []*NetIPNet{},
+			AllowNondistributableArtifactsHostnames: []string{},
+		})
+		assert.NilError(t, err)
+		const expected = `{"AllowNondistributableArtifactsCIDRs":null,"AllowNondistributableArtifactsHostnames":null,"IndexConfigs":null,"InsecureRegistryCIDRs":null,"Mirrors":null}`
+		assert.Check(t, is.Equal(string(b), expected))
+	})
+}

cmd/dockerd/config.go

@@ -18,6 +18,7 @@ func installCommonConfigFlags(conf *config.Config, flags *pflag.FlagSet) {
 		insecureRegistries    = opts.NewNamedListOptsRef("insecure-registries", &conf.InsecureRegistries, registry.ValidateIndexName)
 	)
 	flags.Var(allowNonDistributable, "allow-nondistributable-artifacts", "Allow push of nondistributable artifacts to registry")
+	_ = flags.MarkDeprecated("allow-nondistributable-artifacts", "Pushing nondistributable artifacts is now enabled by default. ")
 	flags.Var(registryMirrors, "registry-mirror", "Preferred Docker registry mirror")
 	flags.Var(insecureRegistries, "insecure-registry", "Enable insecure registry communication")
 

cmd/dockerd/daemon.go

@@ -653,6 +653,11 @@ func loadDaemonCliConfig(opts *daemonOptions) (*config.Config, error) {
 		return nil, err
 	}
 
+	if len(conf.AllowNondistributableArtifacts) > 0 {
+		// TODO(thaJeztah): move to config.Validate and change into an error for v29.0 and remove in v30.0.
+		log.G(context.TODO()).Warn(`DEPRECATED: The "allow-nondistributable-artifacts" config parameter is deprecated and always enabled; this option will be removed in the next release`)
+	}
+
 	return conf, nil
 }
 

cmd/dockerd/daemon_test.go

@@ -190,7 +190,6 @@ func TestLoadDaemonConfigWithEmbeddedOptions(t *testing.T) {
 
 func TestLoadDaemonConfigWithRegistryOptions(t *testing.T) {
 	content := `{
-		"allow-nondistributable-artifacts": ["allow-nondistributable-artifacts.example.com"],
 		"registry-mirrors": ["https://mirrors.example.com"],
 		"insecure-registries": ["https://insecure-registry.example.com"]
 	}`
@@ -202,7 +201,6 @@ func TestLoadDaemonConfigWithRegistryOptions(t *testing.T) {
 	assert.NilError(t, err)
 	assert.Assert(t, loadedConfig != nil)
 
-	assert.Check(t, is.Len(loadedConfig.AllowNondistributableArtifacts, 1))
 	assert.Check(t, is.Len(loadedConfig.Mirrors, 1))
 	assert.Check(t, is.Len(loadedConfig.InsecureRegistries, 1))
 }

daemon/reload.go

@@ -231,10 +231,6 @@ func (daemon *Daemon) reloadLabels(txn *reloadTxn, newCfg *configStore, conf *co
 // reloadRegistryConfig updates the configuration with registry options
 // and updates the passed attributes.
 func (daemon *Daemon) reloadRegistryConfig(txn *reloadTxn, newCfg *configStore, conf *config.Config, attributes map[string]string) error {
-	// Update corresponding configuration.
-	if conf.IsValueSet("allow-nondistributable-artifacts") {
-		newCfg.ServiceOptions.AllowNondistributableArtifacts = conf.AllowNondistributableArtifacts
-	}
 	if conf.IsValueSet("insecure-registries") {
 		newCfg.ServiceOptions.InsecureRegistries = conf.InsecureRegistries
 	}
@@ -248,7 +244,6 @@ func (daemon *Daemon) reloadRegistryConfig(txn *reloadTxn, newCfg *configStore,
 	}
 	txn.OnCommit(func() error { commit(); return nil })
 
-	attributes["allow-nondistributable-artifacts"] = marshalAttributeSlice(newCfg.ServiceOptions.AllowNondistributableArtifacts)
 	attributes["insecure-registries"] = marshalAttributeSlice(newCfg.ServiceOptions.InsecureRegistries)
 	attributes["registry-mirrors"] = marshalAttributeSlice(newCfg.ServiceOptions.Mirrors)
 

daemon/reload_test.go

@@ -2,7 +2,6 @@ package daemon // import "github.com/docker/docker/daemon"
 
 import (
 	"os"
-	"sort"
 	"testing"
 
 	"github.com/containerd/log"
@@ -11,7 +10,6 @@ import (
 	"github.com/docker/docker/libnetwork"
 	"github.com/docker/docker/registry"
 	"gotest.tools/v3/assert"
-	is "gotest.tools/v3/assert/cmp"
 )
 
 // muteLogs suppresses logs that are generated during the test
@@ -62,61 +60,6 @@ func TestDaemonReloadLabels(t *testing.T) {
 	}
 }
 
-func TestDaemonReloadAllowNondistributableArtifacts(t *testing.T) {
-	daemon := newDaemonForReloadT(t, &config.Config{})
-	muteLogs(t)
-
-	var err error
-	// Initialize daemon with some registries.
-	daemon.registryService, err = registry.NewService(registry.ServiceOptions{
-		AllowNondistributableArtifacts: []string{
-			"127.0.0.0/8",
-			"10.10.1.11:5000",
-			"10.10.1.22:5000", // This will be removed during reload.
-			"docker1.com",
-			"docker2.com", // This will be removed during reload.
-		},
-	})
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	registries := []string{
-		"::1/128",
-		"127.0.0.0/8",
-		"10.10.1.11:5000",
-		"10.10.1.33:5000", // This will be added during reload.
-		"docker1.com",
-		"docker3.com", // This will be added during reload.
-	}
-
-	newConfig := &config.Config{
-		CommonConfig: config.CommonConfig{
-			ServiceOptions: registry.ServiceOptions{
-				AllowNondistributableArtifacts: registries,
-			},
-			ValuesSet: map[string]interface{}{
-				"allow-nondistributable-artifacts": registries,
-			},
-		},
-	}
-
-	if err := daemon.Reload(newConfig); err != nil {
-		t.Fatal(err)
-	}
-
-	var actual []string
-	serviceConfig := daemon.registryService.ServiceConfig()
-	for _, value := range serviceConfig.AllowNondistributableArtifactsCIDRs {
-		actual = append(actual, value.String())
-	}
-	actual = append(actual, serviceConfig.AllowNondistributableArtifactsHostnames...)
-
-	sort.Strings(registries)
-	sort.Strings(actual)
-	assert.Check(t, is.DeepEqual(registries, actual))
-}
-
 func TestDaemonReloadMirrors(t *testing.T) {
 	daemon := &Daemon{
 		imageService: images.NewImageService(images.ImageServiceConfig{}),

docs/api/version-history.md

@@ -17,6 +17,11 @@ keywords: "API, Docker, rcli, REST, documentation"
 
 [Docker Engine API v1.48](https://docs.docker.com/reference/api/engine/version/v1.48/) documentation
 
+# Deprecated: The "allow-nondistributable-artifacts" daemon configuration is
+  deprecated and enabled by default. The  `AllowNondistributableArtifactsCIDRs`
+  and `AllowNondistributableArtifactsHostnames` fields in the `RegistryConfig`
+  struct in the `GET /info` response will now always be `null` and will be
+  omitted in API v1.49.
 * `GET /images/{name}/history` now supports a `platform` parameter (JSON
   encoded OCI Platform type) that allows to specify a platform to show the
   history of.