indiandataclub

Enrolled into Databricks challenge to build a strong foundation Databricks through daily learning, hands-on practice, and real-world problem solving. This Challenge is Sponsored by Databricks and organised by Codebasics(https://codebasics.io/) and Indian Data Club(https://www.indiandataclub.com/).

End-to-End workforce Intelligence Project ~1.3M LinkedIn job postings from 2024. Implements a Bronze–Silver–Gold architecture in Databricks to extract confidence-aware role and skill demand signals. Focuses on skill penetration, persistence, role skill structure, geographic variation, and data coverage to support transparent labor market analysis.

This Repository contains the Notebooks and Code from Day 13 of the IDC Databricks 14 Days AI Challenge. It focuses on training and comparing multiple models, hyperparameter tuning, building Spark ML pipelines, evaluating runs in MLflow, and selecting an appropriate model based on metrics, simplicity, and interpretability.

This repository contains the notebooks, code, and outputs from Day 11 of the IDC Databricks 14 Days AI Challenge. It focuses on descriptive statistics, hypothesis testing, correlation analysis, and feature engineering, capturing insights from event-level data and preparing features for downstream machine learning.

This repository captures hands-on work from Day 3 and Day 4 of the IDC Databricks 14 Days AI Challenge. It covers PySpark fundamentals, window functions, feature engineering, and Delta Lake concepts including ACID transactions, schema enforcement, and SQL-based Delta table creation using an e-commerce dataset.

Day-14 Final Production System from the Databricks 14 Days AI Challenge – 2. Integrated the data pipeline and ML pipeline, trained a Logistic Regression Model, Logged the Model with MLflow, Generated Purchase Predictions through Batch Inference, & Stored Results in a Gold Delta Table.

Day 11 of Phase 3: Performance & Production Thinking from the Databricks 14 Days AI Challenge – 2 (Advanced). Demonstrated Delta Lake Time Travel by Appending Records, Querying Historical Table Versions, Comparing Dataset States, & Exploring Retention Limitations affecting Historical Data Recovery.

Day 10 of Phase 3: Performance & Production Thinking from the Databricks 14 Days AI Challenge – 2 (Advanced). Executed Heavy Analytical Queries, Analyzed Spark Explain Plans, Compared Execution Times, & Observed Query Optimization Effects while Working within Serverless Compute Limitations.

Day-13 Architecture Design: Carousel Pages & Pipeline Diagram from Insights across Day-0–Day-12.

This repository contains hands-on work from Day 7 and Day 8 of the IDC Databricks 14 Days AI Challenge. It covers Databricks Jobs, multi-task workflows, parameters, scheduling, and error handling, along with Unity Catalog concepts such as catalogs, schemas, table registration, and governed views.

DATABRICKS 14 DAYS AI CHALLENGE

A 21-day journey from SQL beginner to advanced — daily concepts, practice queries, and challenges.

Day 1 of Phase 1: Better Data Engineering from the Databricks 14 Days AI Challenge – 2 (Advanced). Covers CSV to Delta conversion, Managed Table Creation, Incremental Appends, Small File Simulation, OPTIMIZE Analysis, and Metadata-driven Reasoning using DESCRIBE DETAIL.

This repository contains the notebooks, SQL queries, and outputs from Day 14 of the IDC Databricks 14 Days AI Challenge. It focuses on AI-assisted analytics using Databricks Genie and Mosaic AI, including natural language query, sentiment analysis, data joins, visualizations, and AI-supported insight generation.

This repository documents my learning journey with Databricks as part of the IDC Databricks 14 Days AI Challenge. It includes notebooks and code covering platform setup, workspace navigation, compute basics, and introductory PySpark concepts, focused on building strong foundations.

This repository contains the notebooks and code from Day 12 of the IDC Databricks 14 Days AI Challenge. It focuses on MLflow fundamentals, including experiment tracking, logging parameters and metrics, model logging, comparing runs in the MLflow UI, and understanding evaluation results across different test sizes.

This repository contains the notebooks and code from Day 10 of the IDC Databricks 14 Days AI Challenge. It focuses on Spark performance tuning, including query execution plan analysis, partitioning strategies, OPTIMIZE and ZORDER usage, caching techniques, and benchmarking improvements in Databricks.

Day 8 of Phase 2: AI System Building from the Databricks 14 Days AI Challenge – 2 (Advanced). Implemented Batch Inference to Score 5.3M+ Users, Extracted Class Probabilities from Spark VectorUDT, persisted Results to a Gold Delta Table, and ranked Top Predicted Buyers while resolving Schema Nuances.

This repository contains hands-on work from Day 5 and Day 6 of the IDC Databricks 14 Days AI Challenge. It covers Delta Lake advanced operations such as time travel, MERGE, and OPTIMIZE, along with designing and implementing a Bronze–Silver–Gold architecture using incremental processing patterns.

This Repository Documents Day 0 (Prerequisite) of the 14 Days Databricks AI Challenge – 2 (Advanced). It covers Databricks Environment Setup, Kaggle API Configuration, Schema and Volume Creation, and loading an E-Commerce Dataset into the Lakehouse, following the Official Roadmap.