feat(mlop-2760): Create metadata module (feature, feature_set, reader, writer and pipeline)

[albjoaov]

Why

This change introduces a robust metadata generation system for Butterfree components. The primary motivation is to create a standardized and automated way to:

• Document the schema of feature set pipelines and their components.
• Facilitate serialization of pipeline metadata for various use cases, such as cataloging, lineage tracking, and integration with other tools.

This will improve the maintainability, understandability, and interoperability of feature set pipelines within the Butterfree ecosystem.

What

New metadata module:

- Introduced a new `butterfree/metadata` directory to house all Pydantic models for metadata.

Dependencies

 - `Pydantic` added to requirements

Tests

- Unit tests for EVERY behavior of new class
- Integration tests for pipeline with `FeatureSet` and with `AggregatedFeatureSet`

How

The solution refactors and centralizes metadata definition using Pydantic models within a dedicated butterfree/metadata directory.

1. Hierarchical Metadata Construction:

   • Individual components (Feature, KeyFeature, TimestampFeature, Window, Reader subclasses, Writer) are responsible for building their own metadata Pydantic models (FeatureMetadata, ReaderMetadata variants, WriterMetadata).

   • Container classes (FeatureSet, AggregatedFeatureSet, FeatureSetPipeline) aggregate metadata from their constituent parts. For instance, FeatureSet.build_metadata() calls build_metadata() on its keys, timestamp, and then constructs metadata for its transformed features before packaging it all into a FeatureSetMetadata object.

2. Pydantic Model Usage: Each metadata type (e.g., for a feature, a reader, a feature set) has a corresponding Pydantic model ensuring structure and validation.

[homesbot]

🎉 Snyk checks have passed. No issues have been found so far.

✅ code/snyk check is complete. No issues have been found. (View Details)

[lecardozo]

nice work! I have some minor questions:

• We have this butterfree/reports/metadata.py logic that is used behind the scenes for building the markdown catalog. Do you think we would benefit from that?
• If we decide on replacing that abstraction with this pydantic-based one, I think we can maybe have each component generating its own "serialized" version so the "parent" component would call the children build_metadata method

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud