Installing and Configuring Apache Airflow for ML Workflows

Description

This project sets up Apache Airflow to automate and manage an ML pipeline, covering stages from data ingestion to model deployment. The workflow is designed as a DAG to ensure seamless task execution and monitoring.

Utilities Used

• Apache Airflow
• Python 3.8
• Linux (Ubuntu 20.04)
• Airflow Web UI

Project Walk-through

Set Up Airflow


Initialize Airflow DB and create an admin user


Start Web Server and Scheduler


Create ML Workflow DAG


Run and Monitor the DAG