Skip to content

Add Best Practices Section to Documentation #27

@Mmodarre

Description

@Mmodarre

Overview

Add a comprehensive "Best Practices" section to the Lakehouse Plumber documentation to help users follow recommended patterns and conventions when building data pipelines.

Motivation

As the LHP framework grows, users would benefit from documented best practices that promote:

  • Code reusability and maintainability
  • Consistent project structure
  • Efficient use of framework features
  • Better collaboration across teams

Proposed Content

The Best Practices section should cover (at minimum):

1. Template Usage

  • Use templates as much as possible for common pipeline patterns
  • Benefits of template-based approach (DRY principle, easier maintenance, standardization)
  • When to create custom templates vs. using existing ones
  • Examples of template reuse across different data sources

2. Jinja Features in Actions

  • Leverage Jinja conditions and loops in action definitions
  • Examples of conditional logic in pipeline configurations
  • Using loops to reduce repetitive action definitions
  • Variable substitution and environment-specific configurations
  • Best practices for keeping Jinja logic readable and maintainable

3. File Naming Conventions

  • Add template identifiers to file names for easy identification
  • Recommended naming patterns (e.g., <entity>_<template_id>.yaml)
  • How naming conventions improve project navigation and understanding
  • Examples from standard templates (TMPL001-TMPL009)
  • Organizing files by pipeline stage (raw ingestion, bronze, silver, etc.)

Additional Considerations

  • Include practical examples from real-world usage
  • Add cross-references to relevant template documentation
  • Consider adding a "Common Patterns" subsection
  • Include anti-patterns or things to avoid

Acceptance Criteria

  • Best Practices section added to documentation
  • All three topics (templates, Jinja, naming) covered with examples
  • Documentation is linked from main README or documentation index
  • Examples are tested and verified to work with current LHP version

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentation

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions