The concern of the difference between the deployed eoapi on the cluster and the architecture depicted in the 'Default Configuration' graph + How to manage large-scale pgstac database?

[MathewNWSH]

Hi, I'm a kubernetes newbie starting my journey with eoapi and STAC. I'm trying to create a STAC catalog for my organization containing data from various Copernicus program missions. I managed to create a sample catalog using pgstac and stac-fastapi-pgstac with Docker Compose. Now, due to the potential scalability of FastAPI, I would like to try doing this on Kubernetes, and that's how I came across eoapi :)

Using the command:

helm install -n eoapi --create-namespace eoapi eoapi/eoapi --version 0.2.14 -f config.yaml

I managed to deploy eoapi on my cluster:

(base) mniemyjski@mniemyjskiNB:~$ kubectl get all -n eoapi
NAME                                READY   STATUS    RESTARTS   AGE
pod/doc-server-7c84dcb5c7-q94tx     1/1     Running   0          49s
pod/pgstac-5b5c44dfc6-z5nhk         0/2     Pending   0          49s
pod/raster-eoapi-684887cf97-5k6cg   1/1     Running   0          49s
pod/stac-eoapi-c575676c4-6kkcr      1/1     Running   0          49s
pod/vector-eoapi-69fb797f4f-zz8gv   1/1     Running   0          49s

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/doc-server   ClusterIP   10.254.240.170   <none>        80/TCP     77s
service/pgstac       ClusterIP   10.254.199.172   <none>        5432/TCP   85s
service/raster       ClusterIP   10.254.175.254   <none>        8080/TCP   85s
service/stac         ClusterIP   10.254.233.78    <none>        8080/TCP   66s
service/vector       ClusterIP   10.254.97.109    <none>        8080/TCP   83s

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/doc-server     1/1     1            1           64s
deployment.apps/pgstac         0/1     1            0           64s
deployment.apps/raster-eoapi   1/1     1            1           64s
deployment.apps/stac-eoapi     1/1     1            1           64s
deployment.apps/vector-eoapi   1/1     1            1           64s

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/doc-server-7c84dcb5c7     1         1         1       49s
replicaset.apps/pgstac-5b5c44dfc6         1         1         0       49s
replicaset.apps/raster-eoapi-684887cf97   1         1         1       49s
replicaset.apps/stac-eoapi-c575676c4      1         1         1       49s
replicaset.apps/vector-eoapi-69fb797f4f   1         1         1       49s

However, looking at the documentation, it seemed to me that as part of the Helm installation, a service load balancer and an nginx service would also be provided? That is:

To access STAC externally, I changed the deployment type of service/stac from Cluster IP to LoadBalancer, which assigned an external IP address to the service. But considering the presented graph, I should rather be accessing the catalog through:

http://<loadbalancer external IP>/stac
rather than what I'm currently doing:
http://<service/stac external IP>:8080/

My first question is how to achieve the situation depicted on the graph?

The next question is more forward-thinking, for the time when I've sorted out the first one ^^ Specifically, can I connect eoapi to an external pgstac database? If so, how? The catalog I'll be attempting to create corresponds to a size of many petabytes of EO data stored in a database of around 1 terabyte. I know that pgstac has internal optimizations to ensure quick responses to STAC queries (thanks to partitioning, indexing). Therefore, I was thinking of creating something like this:

Unless this idea is unnecessary, and pgstac and eoapi can handle a database of this size, with the only limitation being the size of the node?

thanks,
Mat

[vincentsarago]

@MathewNWSH have you looked at https://github.com/developmentseed/eoapi-k8s

[MathewNWSH]

Yes of course, but skipped the part responsible for creating cluster. Here are the answers to the first question:
https://github.com/developmentseed/eoapi-k8s/blob/main/docs/aws-eks.md#install-aws-load-balancer-controller-
https://github.com/developmentseed/eoapi-k8s/blob/main/docs/aws-eks.md#install-nginx-ingress-controller-
sorry for silly overlook!

[MathewNWSH]

Okay i get it now that cluster configured on AWS needs to be set up with some not default settings - get it. I'm using magnum to launch clusters on CREODIAS cloud and most of this options I have already enabled by default.

Also found the ingress on cluster:

so the last thing (right now) I'm missing is the LoadBalancer.

Also found the answer for second question here -> developmentseed/eoapi-k8s#65

[MathewNWSH]

Yup, I skipped the part responsible for creation of nginx ingress-controler. Still struggling to understand the way k8s wroks :)
Here is the instruction on how to launch k8s magnum cluster on CREODIAS, which (at least it seems) works just fine with eoapi:
Log to: https://horizon.cloudferro.com
Move to the Clusters tab on the left side of the dashboard:

Hit create cluster button on the right side:

This is the configuration that worked for me:

(probably also works for smaller flavours)



and then hit submit.

after few minutes cluster starts, config.yaml can be accessed using Show Cluster Config