Separating Server API from the remainder

[Guilherme-B]

Hey team! 👋

I have been exploring the idea of using Airbyte instead of Dlt. And unfortunately, PyAirbyte is just not very mature yet.

A critical topic is cost, what I've been thinking is having the Server / API - to add connections, test them, etc - in a constantly live cluster, and then having the main components (webui, temporal service, worker, workload api, workload launcher, cron and bootloader) in an environment which is turned on or off via an Orchestrator.

I was looking at abctl but as far as I can see, the whole thing is combined. Anyone tried this architecture?