-
Notifications
You must be signed in to change notification settings - Fork 8
Open
Labels
documentationImprovements or additions to documentationImprovements or additions to documentation
Description
Overview
Add a comprehensive "Best Practices" section to the Lakehouse Plumber documentation to help users follow recommended patterns and conventions when building data pipelines.
Motivation
As the LHP framework grows, users would benefit from documented best practices that promote:
- Code reusability and maintainability
- Consistent project structure
- Efficient use of framework features
- Better collaboration across teams
Proposed Content
The Best Practices section should cover (at minimum):
1. Template Usage
- Use templates as much as possible for common pipeline patterns
- Benefits of template-based approach (DRY principle, easier maintenance, standardization)
- When to create custom templates vs. using existing ones
- Examples of template reuse across different data sources
2. Jinja Features in Actions
- Leverage Jinja conditions and loops in action definitions
- Examples of conditional logic in pipeline configurations
- Using loops to reduce repetitive action definitions
- Variable substitution and environment-specific configurations
- Best practices for keeping Jinja logic readable and maintainable
3. File Naming Conventions
- Add template identifiers to file names for easy identification
- Recommended naming patterns (e.g.,
<entity>_<template_id>.yaml) - How naming conventions improve project navigation and understanding
- Examples from standard templates (TMPL001-TMPL009)
- Organizing files by pipeline stage (raw ingestion, bronze, silver, etc.)
Additional Considerations
- Include practical examples from real-world usage
- Add cross-references to relevant template documentation
- Consider adding a "Common Patterns" subsection
- Include anti-patterns or things to avoid
Acceptance Criteria
- Best Practices section added to documentation
- All three topics (templates, Jinja, naming) covered with examples
- Documentation is linked from main README or documentation index
- Examples are tested and verified to work with current LHP version
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
documentationImprovements or additions to documentationImprovements or additions to documentation