Data Types mapping from Database Source and Database Destination

[koliankolin]

Hello!

By default there is a mapping mysql Decimal type into float BigQuery. It seems not good. How I can create a mapping from mysql Decimal into BigQuery Numeric? For my case it is critical because we need to maintain precision and load in Numeric from mysql

[marcosmarxm]

Hello @koliankolin, today this is not possible. Currently, the source will transform the record to an AirbyteRecord type (JsonSchema), and then the Destination tries to decode the datatype to their closest format. I think for this situation, it is (decimal) -> (number) -> (float). This is the strategy Airbyte has to allow connections from any type of source to any destination. If instead, you need a specific mapping from MySQL to BigQuery, imagine the complexity it can create among all the sources and destinations Airbyte has in the catalog. @evantahler may have more insights about this topic.

Thanks for the question. I'll change the topic to a more generic one.

[evantahler]

We do intend to eventually expand the data-types Airbyte supports, but likely won't be until next year.

That said, as we update how we create typed columns in all of our destinations (Destinations V2), we will be revisiting the best destination-datatypes for each case. Yes, BigQuery should be using DECIMAL for non-integer numbers, and we will be including that change. Join the Beta if you want to try out the new destination early - we will start the beta program later this month!