Data Engineering for AI/ML - September 12, 2024 - Fully Virtual

[deepyaman]

https://home.mlops.community/public/events/dataengforai

Title

Building the Python-first composable analytics stack

Abstract

SQL has reigned king of the data transformation world, and tools like dbt have formed a cornerstone of the modern data stack. However, the rise of composable data systems combined with the emergence of key open-source technologies over the past few years gives data engineers the power to choose the right interface for them. Now, Ibis can provide the same benefits of SQL execution with a flexible Python dataframe API, and we can leverage it to build scalable Python pipelines in Kedro. dlt leverages the power of Apache Arrow for performant EL workflows, while Pandera (via the Ibis backend and Kedro-Pandera integration) provides fully-integrated data validation using the execution engine of your choice.