You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
-[How much is this going to save us?](#how-much-is-this-going-to-save-us)
23
24
-[Infrastructure Needed](#infrastructure-needed)
24
-
-[Hack on this doc](#hack-on-this-doc)
25
25
<!-- /toc -->
26
26
27
27
## Summary
@@ -32,14 +32,30 @@ The global community should be routed to the appropriate mirror for their countr
32
32
33
33
This KEP should cover the policy and distribution mechanisms we will put in place to allow creating a globally distributed, multi-cloud and country solution.
34
34
35
+
## Background (from wiki)
36
+
35
37
## Motivation
36
38
37
-
Currently we push to a single provider, and distributing to the rest of community comes at great cost nearing $150k/month (mostly egress) in donations.
39
+
For a few years now, we have been using k8s.gcr.io in all our repositories as default repository for downloading images from.
38
40
39
-
Additionally, some of our community members are unable to access the official release artifacts due to country level firewalls that do not them connect to Google services.
41
+
The cost of distributing Kubernetes comes at great cost nearing $150kUSD/month (mostly egress) in donations.
42
+
43
+
Additionally some of our community members are unable to access the official release artifacts due to country level firewalls that do not them connect to Google services.
40
44
41
45
Ideally we can dramatically reduce cost and allow everyone in the world to download the artifacts released by our community.
42
46
47
+
We are now used to using the [image promoter process](https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/1734-k8s-image-promoter) to promote images to the official kubernetes container registry using the infrastructure (GCR staging repos etc) provided by [sig-k8s-infra](https://github.com/kubernetes/k8s.io/tree/main/k8s.gcr.io)
48
+
49
+
## Why a new domain?
50
+
51
+
So far we (all kubernetes project) are using GCP as our default infrastructure provider for all things like GCS, GCR, GKE based prow clusters etc. Google has graciously sponsored a lot of our infrastructure costs as well. However for about a year or so we are finding that our costs are sky-rocketing because the community usage of this infrastructure has been from other cloud providers like AWS, Azure etc. So in conjunction with CNCF staff we are trying to put together a plan to host copies of images and binaries nearer to where they are used rather than incur cross-cloud costs.
52
+
53
+
One part of this plan is to setup a proxy OCI service, that can identify where the traffic is coming from and redirect to the nearest image layer/repository. This is why we are setting up a new service using what we call an [oci-proxy](https://github.com/kubernetes-sigs/oci-proxy) for everyone to use. This proxy will identify traffic coming from, for example, a certain AWS region, then will setup a HTTP redirect to a source in that AWS region. If we get traffic from GKE/GCP or we don't know where the traffic is coming from, it will still redirect to the current infrastructure (k8s.gcr.io).
54
+
55
+
## How can we help?
56
+
57
+
When Kubernetes master opens up for v1.25 development, we need to update all default urls in our code and test harness to the new registry url. As a team sig-k8s-infra is signing up to ensure that this oci-proxy based registry.k8s.io will be as robust and available as the current setup. As a backup, we will continue to run the current k8s.gcr.io as well. So do not worry about that going away. Turning on traffic to the new url will help us monitor and fix things if/when they break and we will be able to tune traffic and lower our costs of operation.
58
+
43
59
### Goals
44
60
45
61
A policy and procedure for use by SIG Release to promote container images and release binaries to multiple registries and mirrors.
@@ -48,7 +64,17 @@ A solution to allow redirection to appropriate mirrors to lower cost and allow a
48
64
49
65
### Non-Goals
50
66
51
-
Anything related to creation of artifacts, bom, digital signatures.
67
+
Anything related to creation of artifacts, bom, digital signatures, staging buckets.
68
+
69
+
### What is not in scope
70
+
71
+
- Currently we focus on AWS only. We are getting a lot of help from AWS in terms of technical details as well as targeted infrastructure costs for standing up and running this infrastructure
72
+
73
+
### What are good goals to shoot for
74
+
75
+
- In terms of cost reduction, monitor GCP infrastructure and get to the point where we fully avoid serving large binary image layers from GCR/GCS
76
+
- We can add other AWS regions and clouds as needed in well known documented way
77
+
- Seamless transition for the community from the old k8s.gcr.io to registry.k8s.io with same rock solid stability as we now have with k8s.gcr.io
52
78
53
79
## Proposal
54
80
@@ -58,6 +84,19 @@ First, the policy and procedures to promote/upload our artifacts to multiple pro
58
84
59
85
Additionally we require a registry and artifact url-redirection solution to the local cloud provider or country.
60
86
87
+
## What exactly are you doing?
88
+
89
+
- We are setting up an AWS account with an IAM role and s3 buckets in AWS regions where we see a large percentage of source image pull traffic
90
+
- We will iterate on a sandbox url (registry.sandbox.k8s.io) for our experiments and ONLY promote things to (registry.k8s.io) when we have complete confidence
91
+
- both registry and registry.sandbox are serving traffic using oci-proxy on google cloud run
92
+
- oci-proxy will be updated to identify incoming traffic from AWS regions based on IP ranges so we can route traffic to s3 buckets in that region. If a specific AWS region do not currently host s3 buckets, we will redirect to the nearest region which does have s3 buckets (tradeoff between storage and network costs)
93
+
- We will bulk sync existing image layers to these s3 layers as a starting point (from GCS/GCR)
94
+
- We will update image-promoter to push to these s3 buckets as well in addition to the current setup
95
+
- We will set up monitoring/reporting to check on new costs we incur on the AWS infrastructure and update what we do in GCP infrastructure as well to include the new components
96
+
- We will have a plan in place on how we could add additional AWS regions in the future
97
+
- We will have CI jobs that will run against registry.sandbox as well to monitor stability before we promote code to registry
98
+
- We will automate the deployment/monitoring and testing of code landing in the oci-proxy repository
99
+
61
100
### User Stories
62
101
63
102
#### SIG Release - Image Promotion
@@ -94,7 +133,7 @@ This is the primary pipeline for delivering Kubernetes worldwide. Ensuring the a
94
133
95
134
## Design Details
96
135
97
-
### Artifact Promotion
136
+
### Release Promotion
98
137
99
138
#### Policy
100
139
@@ -112,24 +151,33 @@ Currently the promotion process is primarily driven by the CIP/[promo-tool#kprom
172
+
Cost of K8s Artifact hosting - Data Studio Graphs
173
+
174
+
128
175
129
176
## Infrastructure Needed
130
177
131
-
It would be good to request some donations for some larger providers, including one in China, via [Cloud Native Credits program](https://www.cncf.io/credits/).
178
+
It would be good to request some donations for some larger providers, including one in China, via cncf.io/credits
0 commit comments