diff --git a/README.md b/README.md index acfa776..8793606 100644 --- a/README.md +++ b/README.md @@ -7,10 +7,37 @@ [![Python](https://img.shields.io/badge/Python-3.8+-3776AB?logo=python)](https://www.python.org/) [![GitHub Actions](https://img.shields.io/badge/CI/CD-GitHub%20Actions-2088FF?logo=github-actions)](https://github.com/features/actions) +## 🌐 GitHub Pages + +The GitHub Pages for this repository is available at: **[Delta Lake & Apache Iceberg Knowledge Hub](https://analytical-guide.github.io/Datalake-Guide/)** + ## 🎯 Vision Statement **Building the definitive, community-driven knowledge ecosystem for modern data lakehouse technologies.** This repository serves as a living, breathing whitepaper that evolves with the data engineering landscape, combining comprehensive technical comparisons, battle-tested code recipes, and AI-powered content curation to empower data engineers worldwide to make informed architectural decisions and implement best practices for Delta Lake and Apache Iceberg. +## 📁 Repository Content and Structure + +This repository is organized into the following sections: + +### Core Content + +| Section | Location | Description | +|---------|----------|-------------| +| **Feature Matrix** | [`docs/comparisons/feature-matrix.md`](docs/comparisons/feature-matrix.md) | Comprehensive comparison of Delta Lake vs Apache Iceberg | +| **Code Recipes** | [`code-recipes/`](code-recipes/) | Production-ready code examples with validation | +| **Tutorials** | [`docs/tutorials/`](docs/tutorials/) | Step-by-step guides for common use cases | +| **Architecture** | [`docs/architecture/`](docs/architecture/) | Reference architectures and design patterns | +| **Best Practices** | [`docs/best-practices/`](docs/best-practices/) | Industry-tested patterns and recommendations | + +### Learning Resources + +| Resource | Location | Description | +|----------|----------|-------------| +| **Getting Started** | [`docs/tutorials/getting-started.md`](docs/tutorials/getting-started.md) | Quick start guide for beginners | +| **Migration Guide** | [`docs/tutorials/migration-guide.md`](docs/tutorials/migration-guide.md) | Moving from legacy systems | +| **Knowledge Quiz** | [`quiz/`](quiz/) | Test your Delta Lake & Iceberg knowledge | +| **Design System** | [`docs/design-system.md`](docs/design-system.md) | UI/UX guidelines for the project | + ## 📚 Quick Links - [🔍 **Feature Comparison Matrix**](docs/comparisons/feature-matrix.md) - Detailed side-by-side comparison of Delta Lake vs Apache Iceberg @@ -72,6 +99,19 @@ Every recipe in our [code-recipes](code-recipes/) directory follows a standardiz - **Best Practices**: Industry-tested patterns and anti-patterns - **Architecture Guides**: Reference implementations for various scales +## 🚀 How to Use This Material + +1. **Start with the Feature Comparison**: Begin by reading the [Feature Comparison Matrix](docs/comparisons/feature-matrix.md) for a comprehensive overview of Delta Lake vs Apache Iceberg. + +2. **Explore the Getting Started Guide**: Use the [Getting Started Tutorial](docs/tutorials/getting-started.md) to set up your first lakehouse. + +3. **Review Code Recipes**: Work through the [Code Recipes](code-recipes/) for hands-on implementation examples. + +4. **Follow Best Practices**: Study the [Best Practices](docs/best-practices/) for production-ready implementations. + +5. **Test Your Knowledge**: Take the [Knowledge Quiz](quiz/) to validate your understanding. + +6. **Visit the Website**: Explore the full content at [GitHub Pages](https://analytical-guide.github.io/Datalake-Guide/). ## 🚀 Getting Started @@ -88,7 +128,7 @@ Every recipe in our [code-recipes](code-recipes/) directory follows a standardiz 3. Review the [Code of Conduct](CODE_OF_CONDUCT.md) 4. Submit your first pull request! -## �️ Development & Deployment +## 🛠️ Development & Deployment ### Prerequisites @@ -291,7 +331,7 @@ Monitor performance using: - **WebPageTest**: External performance testing - **GitHub Actions**: Automated performance checks -## �📈 Repository Stats +## 📈 Repository Stats ![GitHub stars](https://img.shields.io/github/stars/Analytical-Guide/Datalake-Guide?style=social) ![GitHub forks](https://img.shields.io/github/forks/Analytical-Guide/Datalake-Guide?style=social) diff --git a/_config.yml b/_config.yml index 9479b21..373edfa 100644 --- a/_config.yml +++ b/_config.yml @@ -10,6 +10,9 @@ github_username: Analytical-Guide author: "Analytical Guide Community" email: "community@analytical-guide.com" +# Analytics (set to your GA4 measurement ID, e.g., G-XXXXXXXXXX) +google_analytics: "G-LHTGZTRTCX" + # Build settings markdown: kramdown highlighter: rouge @@ -30,6 +33,7 @@ include: - community - _layouts - _includes + - _data - assets # Exclude files diff --git a/_data/navigation.yml b/_data/navigation.yml new file mode 100644 index 0000000..5a171b2 --- /dev/null +++ b/_data/navigation.yml @@ -0,0 +1,80 @@ +- title: "I. Overview" + url: "/" + icon: "fas fa-home" + children: + - title: "Getting Started" + url: "/docs/tutorials/getting-started/" + - title: "Feature Comparison" + url: "/docs/comparisons/feature-matrix/" + +- title: "II. Comparisons" + url: "/docs/comparisons/feature-matrix/" + icon: "fas fa-balance-scale" + children: + - title: "Feature Matrix" + url: "/docs/comparisons/feature-matrix/" + - title: "Time Travel & Versioning" + url: "/docs/comparisons/feature-matrix/#time-travel" + - title: "Schema Evolution" + url: "/docs/comparisons/feature-matrix/#schema-evolution" + +- title: "III. Code Recipes" + url: "/code-recipes/" + icon: "fas fa-code" + children: + - title: "Recipe Catalog" + url: "/code-recipes/" + - title: "Basic Delta Table" + url: "/code-recipes/examples/basic-delta-table/" + - title: "Basic Iceberg Table" + url: "/code-recipes/examples/basic-iceberg-table/" + - title: "Streaming CDC Pipeline" + url: "/code-recipes/examples/streaming-cdc-pipeline/" + - title: "Time Series Forecasting" + url: "/code-recipes/examples/time-series-forecasting/" + +- title: "IV. Tutorials" + url: "/docs/tutorials/" + icon: "fas fa-graduation-cap" + children: + - title: "Tutorials Hub" + url: "/docs/tutorials/" + - title: "Getting Started" + url: "/docs/tutorials/getting-started/" + - title: "Migration Guide" + url: "/docs/tutorials/migration-guide/" + +- title: "V. Architecture" + url: "/docs/architecture/" + icon: "fas fa-cubes" + children: + - title: "Architecture Patterns" + url: "/docs/architecture/" + - title: "System Overview" + url: "/docs/architecture/system-overview/" + - title: "Blueprint" + url: "/docs/BLUEPRINT/" + +- title: "VI. Best Practices" + url: "/docs/best-practices/" + icon: "fas fa-check-circle" + children: + - title: "Best Practices Hub" + url: "/docs/best-practices/" + - title: "Production Readiness" + url: "/docs/best-practices/production-readiness/" + +- title: "VII. Quiz" + url: "/quiz/" + icon: "fas fa-brain" + +- title: "VIII. Community" + url: "/CONTRIBUTING/" + icon: "fas fa-users" + children: + - title: "Contributing" + url: "/CONTRIBUTING/" + - title: "Code of Conduct" + url: "/CODE_OF_CONDUCT/" + - title: "Awesome List" + url: "/docs/awesome-list/" diff --git a/_includes/analytics.html b/_includes/analytics.html new file mode 100644 index 0000000..c47eeb8 --- /dev/null +++ b/_includes/analytics.html @@ -0,0 +1,10 @@ +{% if site.google_analytics and site.google_analytics != "" %} + + + +{% endif %} diff --git a/_includes/sidebar.html b/_includes/sidebar.html new file mode 100644 index 0000000..1c5ecfe --- /dev/null +++ b/_includes/sidebar.html @@ -0,0 +1,26 @@ + diff --git a/_layouts/default.html b/_layouts/default.html index a354424..b4544ae 100644 --- a/_layouts/default.html +++ b/_layouts/default.html @@ -3,44 +3,26 @@ - {% if page.title %}{{ page.title }} | {% endif %}{{ site.title }} - - - + {% seo %} - - - - - - - - - - - - - - - - + {% include analytics.html %} - + - + {% feed_meta %} -
+
@@ -48,23 +30,27 @@
+ + {% unless page.url == '/' %} + {% include sidebar.html %} + {% endunless %} + -
- {% if page.collection == 'docs' %} +
+ {% if page.url contains '/docs/' or page.url contains '/code-recipes/' %}
@@ -102,34 +114,34 @@

Quick Links

-

Community

+

Topics

-

Resources

+

Community

© {{ site.time | date: '%Y' }} {{ site.title }}. Built with ❤️ by the data engineering community.

-

Licensed under Apache License 2.0

+

Licensed under Apache License 2.0

@@ -137,9 +149,215 @@

Resources

- + \ No newline at end of file diff --git a/_layouts/page.html b/_layouts/page.html new file mode 100644 index 0000000..73dae17 --- /dev/null +++ b/_layouts/page.html @@ -0,0 +1,4 @@ +--- +layout: default +--- +{{ content }} diff --git a/assets/css/main.css b/assets/css/main.css index e6074ae..bf73df5 100644 --- a/assets/css/main.css +++ b/assets/css/main.css @@ -1694,4 +1694,445 @@ img[loading="lazy"] { .leaderboard-entry .date { min-width: auto; } +} + +/* ======================================== + SIDEBAR STYLES + ======================================== */ + +.page-wrapper { + display: flex; + flex-direction: column; + min-height: 100vh; +} + +.sidebar { + position: fixed; + top: 70px; + left: 0; + width: 280px; + height: calc(100vh - 70px); + background: var(--color-bg-primary); + border-right: 1px solid var(--color-bg-tertiary); + padding: var(--space-6); + overflow-y: auto; + z-index: 100; + transition: transform var(--transition-base), width var(--transition-base); +} + +.sidebar.collapsed { + transform: translateX(-240px); + width: 40px; +} + +.sidebar h2 { + font-size: var(--text-lg); + font-weight: var(--font-bold); + color: var(--color-text-primary); + margin-bottom: var(--space-6); + padding-bottom: var(--space-3); + border-bottom: 2px solid var(--color-primary); + display: flex; + align-items: center; + gap: var(--space-2); +} + +.sidebar h2 i { + color: var(--color-primary); +} + +.sidebar-toggle { + position: absolute; + top: var(--space-4); + right: var(--space-2); + width: 32px; + height: 32px; + background: var(--color-bg-secondary); + border: 1px solid var(--color-bg-tertiary); + border-radius: var(--radius-full); + display: flex; + align-items: center; + justify-content: center; + cursor: pointer; + transition: all var(--transition-fast); + color: var(--color-text-secondary); + font-size: var(--text-sm); +} + +.sidebar-toggle:hover { + background: var(--color-primary); + color: white; + border-color: var(--color-primary); +} + +.sidebar .nav-list { + list-style: none; + padding: 0; + margin: 0; +} + +.sidebar .nav-list > li { + margin-bottom: var(--space-2); +} + +.sidebar .nav-list > li > a { + display: flex; + align-items: center; + gap: var(--space-2); + padding: var(--space-3) var(--space-4); + color: var(--color-text-secondary); + text-decoration: none; + border-radius: var(--radius-md); + font-weight: var(--font-medium); + transition: all var(--transition-fast); +} + +.sidebar .nav-list > li > a i { + width: 20px; + text-align: center; + color: var(--color-text-tertiary); +} + +.sidebar .nav-list > li > a:hover, +.sidebar .nav-list > li > a.active { + background: var(--color-primary-lighter); + color: var(--color-primary-dark); +} + +.sidebar .nav-list > li > a:hover i, +.sidebar .nav-list > li > a.active i { + color: var(--color-primary); +} + +.sidebar .nav-list > li.active > a { + background: var(--color-primary-lighter); + color: var(--color-primary-dark); + font-weight: var(--font-semibold); +} + +.sidebar .nav-children { + list-style: none; + padding-left: var(--space-6); + margin-top: var(--space-2); +} + +.sidebar .nav-children li a { + display: block; + padding: var(--space-2) var(--space-3); + color: var(--color-text-tertiary); + text-decoration: none; + font-size: var(--text-sm); + border-radius: var(--radius-sm); + transition: all var(--transition-fast); + border-left: 2px solid transparent; +} + +.sidebar .nav-children li a:hover { + color: var(--color-primary); + background: var(--color-bg-secondary); + border-left-color: var(--color-primary-light); +} + +.sidebar .nav-children li a.active { + color: var(--color-primary); + background: var(--color-bg-secondary); + border-left-color: var(--color-primary); + font-weight: var(--font-medium); +} + +/* Main content when sidebar is present */ +.main-content.with-sidebar { + margin-left: 280px; + transition: margin-left var(--transition-base); +} + +.main-content.with-sidebar.sidebar-collapsed { + margin-left: 40px; +} + +/* ======================================== + SEARCH STYLES + ======================================== */ + +.search-container { + position: relative; + display: flex; + align-items: center; + margin-right: var(--space-3); +} + +#search-input { + padding: var(--space-2) var(--space-8) var(--space-2) var(--space-4); + border: 2px solid var(--color-bg-tertiary); + border-radius: var(--radius-lg); + background: var(--color-bg-primary); + color: var(--color-text-primary); + font-size: var(--text-sm); + width: 200px; + transition: all var(--transition-fast); +} + +#search-input:focus { + outline: none; + border-color: var(--color-primary); + box-shadow: 0 0 0 3px rgba(79, 70, 229, 0.1); + width: 250px; +} + +.search-icon { + position: absolute; + right: var(--space-3); + color: var(--color-text-tertiary); + pointer-events: none; + font-size: var(--text-sm); +} + +.search-results { + position: absolute; + top: 100%; + right: 0; + width: 350px; + max-height: 400px; + background: var(--color-bg-primary); + border: 1px solid var(--color-bg-tertiary); + border-radius: var(--radius-lg); + box-shadow: var(--shadow-xl); + z-index: 1000; + overflow-y: auto; + margin-top: var(--space-2); +} + +.search-results-header { + padding: var(--space-4); + border-bottom: 1px solid var(--color-bg-tertiary); + background: var(--color-bg-secondary); +} + +.search-results-header h4 { + margin: 0; + font-size: var(--text-sm); + font-weight: var(--font-semibold); + color: var(--color-text-primary); +} + +.search-results ul { + list-style: none; + margin: 0; + padding: 0; +} + +.search-results li { + border-bottom: 1px solid var(--color-bg-tertiary); +} + +.search-results li:last-child { + border-bottom: none; +} + +.search-results a { + display: block; + padding: var(--space-4); + text-decoration: none; + color: var(--color-text-primary); + transition: background-color var(--transition-fast); +} + +.search-results a:hover { + background: var(--color-bg-secondary); +} + +.search-results strong { + display: block; + font-weight: var(--font-semibold); + margin-bottom: var(--space-1); + color: var(--color-primary); +} + +.search-results p { + font-size: var(--text-sm); + color: var(--color-text-tertiary); + margin: 0; + line-height: var(--leading-normal); +} + +/* ======================================== + PAGE PROGRESS STYLES + ======================================== */ + +.page-progress { + display: flex; + flex-wrap: wrap; + justify-content: flex-start; + align-items: center; + gap: var(--space-4); + margin-top: var(--space-8); + padding: var(--space-5); + background: var(--color-bg-secondary); + border-radius: var(--radius-lg); + border: 1px solid var(--color-bg-tertiary); +} + +.page-time, +.page-difficulty { + display: inline-flex; + align-items: center; + gap: var(--space-2); + font-size: var(--text-sm); + font-weight: var(--font-medium); + padding: var(--space-2) var(--space-3); + border-radius: var(--radius-md); + background: var(--color-bg-primary); +} + +.page-time { + color: var(--color-text-secondary); +} + +.page-time i { + color: var(--color-info); +} + +.page-difficulty.beginner { + color: var(--color-success); + background: rgba(16, 185, 129, 0.1); +} + +.page-difficulty.intermediate { + color: var(--color-warning); + background: rgba(245, 158, 11, 0.1); +} + +.page-difficulty.advanced { + color: var(--color-error); + background: rgba(239, 68, 68, 0.1); +} + +#mark-complete-btn { + padding: var(--space-3) var(--space-5); +} + +#mark-complete-btn.completed { + background: var(--color-success); + border-color: var(--color-success); +} + +#mark-complete-btn.completed:hover { + filter: brightness(0.9); +} + +/* ======================================== + BREADCRUMB STYLES + ======================================== */ + +.breadcrumb { + padding: var(--space-4) var(--space-6); + font-size: var(--text-sm); + color: var(--color-text-tertiary); + background: var(--color-bg-secondary); + border-bottom: 1px solid var(--color-bg-tertiary); + display: flex; + align-items: center; + flex-wrap: wrap; + gap: var(--space-1); +} + +.breadcrumb a { + color: var(--color-primary); + text-decoration: none; + transition: color var(--transition-fast); +} + +.breadcrumb a:hover { + color: var(--color-primary-dark); + text-decoration: underline; +} + +.breadcrumb-separator { + color: var(--color-text-tertiary); + margin: 0 var(--space-1); +} + +/* ======================================== + RESPONSIVE SIDEBAR + ======================================== */ + +@media (max-width: 1024px) { + .sidebar { + transform: translateX(-100%); + width: 280px; + z-index: 1001; + } + + .sidebar.active { + transform: translateX(0); + } + + .sidebar.collapsed { + transform: translateX(-100%); + } + + .main-content.with-sidebar { + margin-left: 0; + } + + .main-content.with-sidebar.sidebar-collapsed { + margin-left: 0; + } +} + +@media (max-width: 768px) { + .search-container { + display: none; + } + + .nav-actions { + gap: var(--space-2); + } + + .page-progress { + flex-direction: column; + align-items: stretch; + } + + .page-progress .btn { + width: 100%; + justify-content: center; + } +} + +/* ======================================== + DARK MODE ENHANCEMENTS + ======================================== */ + +[data-theme="dark"] .sidebar { + background: var(--color-bg-primary); + border-right-color: rgba(255, 255, 255, 0.1); +} + +[data-theme="dark"] .sidebar h2 { + border-bottom-color: var(--color-primary); +} + +[data-theme="dark"] #search-input { + background: var(--color-bg-secondary); + border-color: rgba(255, 255, 255, 0.1); +} + +[data-theme="dark"] #search-input:focus { + border-color: var(--color-primary); +} + +[data-theme="dark"] .search-results { + background: var(--color-bg-secondary); + border-color: rgba(255, 255, 255, 0.1); +} + +[data-theme="dark"] .page-progress { + background: var(--color-bg-secondary); + border-color: rgba(255, 255, 255, 0.1); +} + +[data-theme="dark"] .breadcrumb { + background: var(--color-bg-secondary); + border-bottom-color: rgba(255, 255, 255, 0.1); } \ No newline at end of file diff --git a/docs/best-practices/production-readiness.md b/docs/best-practices/production-readiness.md index 427953b..c40b3b7 100644 --- a/docs/best-practices/production-readiness.md +++ b/docs/best-practices/production-readiness.md @@ -2,6 +2,8 @@ title: Production Readiness for Delta Lake and Apache Iceberg description: Detailed checklist covering data organization, performance, operations, and governance for production deployments. permalink: /docs/best-practices/production-readiness/ +estimated_time: 25 min +difficulty: Advanced --- # Production Readiness for Delta Lake and Apache Iceberg diff --git a/docs/comparisons/feature-matrix.md b/docs/comparisons/feature-matrix.md index 078e235..d402f80 100644 --- a/docs/comparisons/feature-matrix.md +++ b/docs/comparisons/feature-matrix.md @@ -2,6 +2,8 @@ title: Delta Lake vs Apache Iceberg Feature Comparison Matrix description: Comprehensive side-by-side comparison of capabilities across Delta Lake and Apache Iceberg. permalink: /docs/comparisons/feature-matrix/ +estimated_time: 20 min +difficulty: Beginner --- # Delta Lake vs Apache Iceberg: Feature Comparison Matrix diff --git a/docs/tutorials/getting-started.md b/docs/tutorials/getting-started.md index 54b49d9..48e4a0c 100644 --- a/docs/tutorials/getting-started.md +++ b/docs/tutorials/getting-started.md @@ -2,6 +2,8 @@ title: Getting Started with Delta Lake and Apache Iceberg description: Beginner-friendly tutorial that explains how to stand up and work with both table formats. permalink: /docs/tutorials/getting-started/ +estimated_time: 30 min +difficulty: Beginner --- # Getting Started with Delta Lake and Apache Iceberg diff --git a/docs/tutorials/migration-guide.md b/docs/tutorials/migration-guide.md index 2039aab..742ed5b 100644 --- a/docs/tutorials/migration-guide.md +++ b/docs/tutorials/migration-guide.md @@ -2,6 +2,8 @@ title: Data Lake Migration Guide description: Strategies and checklists for migrating Parquet lakes to Delta Lake or Apache Iceberg with minimal risk. permalink: /docs/tutorials/migration-guide/ +estimated_time: 45 min +difficulty: Intermediate --- # Data Lake Migration Guide: From Parquet to Delta Lake & Apache Iceberg diff --git a/search.json b/search.json new file mode 100644 index 0000000..279c764 --- /dev/null +++ b/search.json @@ -0,0 +1,14 @@ +--- +layout: null +--- +[ + {% for page in site.pages %} + {% if page.title %} + { + "title": {{ page.title | jsonify }}, + "url": {{ page.url | relative_url | jsonify }}, + "content": {{ page.content | strip_html | truncatewords: 100 | jsonify }} + }{% unless forloop.last %},{% endunless %} + {% endif %} + {% endfor %} +]