Move away from that GKE cluster

[rgaiacs]

On 22 September, we were having some problems with the federation redirect hosted on GKE.

On 25 September, Yuvi mentioned on Zulip that

we should move away from that gke cluster somehow

it's really just running fededration-redirect and the analytics stuff and nothing else

[manics]

I don't think there's anything in federation-redirect that requires stored state

mybinder.org-deploy/images/federation-redirect/app.py

Lines 258 to 259 in f10475d

	if self.load_balancer == "rendezvous":
	host_name = rendezvous_rank(self.names_and_weights, cache_key(path))[0]

If so then it should be fine to run it in both federation members. Heztner supports load balancers https://www.hetzner.com/cloud/load-balancer/ , cheapest is €5.39/month
It's not clear to me whether the prime host matters.

[rgaiacs]

At GESIS, there were conversations to move https://gesis.mybinder.org/ to https://binder.gesis.org and apply the GESIS website skin, see #3186.

@arnim is on holiday at the moment and returns to work at the begin of October.

[manics]

and apply the GESIS website skin

Would the work you and @yuvipanda did on decoupling the BinderHub client let us run two BinderHub UIs connecting to the same BinderHub backend, so we could have one with the standard theme and one with the Gesis theme?

[rgaiacs]

The work that was done does not allow to run more than one UI using the same BinderHub backend.

[manics]

I've pulled out our Google Cloud costs for August 2025.

Total: $490.28 (includes $99.56 of "savings")

Split by Service

Service description	Cost ($)	Other savings ($)	Subtotal ($)
Compute Engine	215.85	0	215.85
Cloud SQL	131.91	0	131.91
Kubernetes Engine	148.79	-74.39	74.4
Cloud Storage	43.88	0	43.88
Networking	48.58	-25.16	23.42
Artifact Registry	0.57	0	0.57
Cloud Logging	0.26	0	0.26

Split by cluster name

goog-k8s-cluster-name	Cost ($)	Other savings ($)	Subtotal ($)
goog-k8s-cluster-name:prod	259.39	0	259.39
	227.27	-25.16	202.11
goog-k8s-cluster-name:staging	103.18	-74.39	28.78

227.27 (middle row) is for resources that aren't labelled with a k8s cluster name. This can be further broken down by service into:

Service description	Cost ($)	Other savings ($)	Subtotal ($)
All Cloud SQL cost	131.91	0	131.91
All Cloud Storage cost	43.88	0	43.88
Part of the Networking cost	43.76	25.16	18.60

[manics]

Cloud SQL $131.91

This the MySQL database for Matomo (staging and prod). If we're using plausible we could get rid of it.

SKU description	Usage amount	Usage unit	Cost ($)	Other savings ($)	Subtotal ($)
Cloud SQL for MySQL: Zonal - Extended support 1 vCPU + 3.75GB RAM v57 in Iowa	744	hour	52.08	0	52.08
Cloud SQL for MySQL: Zonal - 1 vCPU + 3.75GB RAM in Americas	744	hour	50.29	0	50.29
Cloud SQL for MySQL: Zonal - Extended support f1-micro v57 in Iowa	744	hour	13.39	0	13.39
Cloud SQL for MySQL: Zonal - Micro instance in Americas	744	hour	7.81	0	7.81
Cloud SQL for MySQL: Zonal - Standard storage in Americas	41	gibibyte month	6.97	0	6.97
Storage PD Snapshot	17.007	gibibyte month	1.36	0	1.36

Cloud Storage $43.88

This is dominated by data transfer costs

SKU description	Usage amount	Usage unit	Cost ($)	Other savings ($)	Subtotal ($)
Network Data Transfer GCP Multi-region within Northern America	1588.424	gibibyte	31.77	0	31.77
Multi-Region Standard Class B Operations	25437336	count	10.17	0	10.17
Multi-Region Standard Class A Operations	107080	count	1.07	0	1.07
Standard Storage US Multi-region	26.268	gibibyte month	0.68	0	0.68
Network Data Transfer GCP Replication within Northern America	9.213	gibibyte	0.18	0	0.18

[yuvipanda]

Let's first kill staging and move that to ovh?

[minrk]

I don't think cloud storage has a reason to go anywhere - we can keep the analytics archive there, unless there's a reason it should move, too? That's a complicated (and necessarily breaking) migration.

Shutting down matomo makes sense. Is there any kind of export we want to do before that, or are we okay with losing pre-plausible data (@choldgraf)? I think of analytics as only short-term data and the events archive as the real long-term data we really want to never lose, so I'm okay with losing matomo.

Moving staging somewhere cheaper makes sense. How much cheaper does it get though, if the subtotal for gke-staging is only $28? Is that savings worth the effort? Prod is likely vastly overprovisioned for just running the federation redirect and analytics archiver.

[yuvipanda]

Yeah we have a stackdriver dependency. I think we should basically move everything else except stackdriver and gcs.

Let’s take a dump of the matomo data and then kill it?

I think the saving will have to come from going 0 on GKE. Let’s run the federation redirect as well as on all our members, and use dns to point to one? Or is that too fraught and we should run like a tiny vm for that?

[minrk]

Let’s take a dump of the matomo data and then kill it?

👍

Let’s run the federation redirect as well as on all our members, and use dns to point to one? Or is that too fraught and we should run like a tiny vm for that?

also 👍 if that works, but I'm not quite sure it does. The federation redirector must run on mybinder.org, which means that every instance must be able to get an ACME certificate for mybinder.org, which only the 'active' one will be able to do, right? Unless we point DNS to all of them at once?

[manics]

If we're using ACME HTTP validation I don't think round-robin DNS will work because you can't ensure the VM that requests the cert will receive the validation request.

I think we'd need to run a separate proxy that terminates HTTPS, cheapest VM is CX23, 2 cpu, 4 GB RAM, 40 GB disk, 20 TB Traffic, € 3.49 /mo.

Hetzner load balancers can terminate SSL but it sounds like you need to delegate a DNS zone to Heztner as it uses DNS validation.