Introducing Table-Level Frequency Times and Connection Groupings in Airbyte

[jahlomp]

I'm a Data Engineer at mPharma, and I've been working extensively with Airbyte. I wanted to bring up a discussion about a couple of features that could potentially enhance Airbyte's functionality, especially for those of us dealing with large databases with numerous tables.

Feature Idea 1: Table-Level Frequency Times
Currently, Airbyte allows us to set synchronization frequencies at the connection level. However, in environments where databases have a large number of tables, each with different update needs, this can be limiting. The idea is to introduce the ability to set synchronization frequencies at the table level. This would provide more granular control, allowing us to optimize syncs based on the specific needs of each table, rather than a one-size-fits-all approach at the connection level.

Challenges & Considerations:

• Managing instance resources: This feature could increase the complexity of how Airbyte manages resources like memory and CPU, as it would potentially need to spin up more containers.
• UI/UX implications: Adjusting the Airbyte interface to accommodate this feature in a user-friendly way.

Feature Idea 2: Grouping Multiple Connections
Another enhancement could be the ability to group multiple connections. This would be particularly useful for managing and organizing connections more efficiently, especially when dealing with a large number of them.

Benefits:

• Improved organization and management of connections.
• Easier monitoring and maintenance of grouped connections.

I believe these features could significantly improve the experience for many Airbyte users, especially in large-scale and complex environments. I understand that these suggestions might have considerable implications for Airbyte's architecture and resource management. Therefore, I'm keen to hear thoughts, feedback, and any potential workarounds or plans regarding these ideas from the community and the Airbyte team.