Refactor Examples & Increase Code Coverage

[bellanov]

With external work ramping up, some efforts can be made in optimizing SerenadeFlow, so it's ready for whatever comes.

It seems the tools will indeed be a collection of plugins of which recipes / examples will be generated. As a result, files like pipeline.py don't seem to serve any real purpose embedded within the source.

We'll work on this one together. I just need to specifically define the concept of examples. Upcoming features will simply require more plugins which will then be accompanied by examples.

Tasks

• Define the concept of examples
• Refactor some of the source functionality as examples (i.e., pipeline.py)
• Increase code coverage
• Start testing examples
• @bellanov to define new directory structure
• @aybanda and @bellanov to clean up existing examples (i.e., FantasyAce), community plugins, and tests wherever necessary
• @aybanda to refactor existing examples into the new directory structure
• @aybanda to architect new examples based on Kaggle datasets

[aybanda]

Hey @bellanov! Completed all the tasks for this issue. Here's what I did:

1. Defined the concept of examples
Created examples/README.md explaining examples as recipes - self-contained scripts that demonstrate ETL patterns. They serve as learning resources, templates, and integration guides.

2. Refactored source functionality as examples

• examples/basic_etl_pipeline.py - core ETL workflow demonstration
• examples/sports_odds_processing.py - domain-specific logic (sports betting validation/flattening) that was previously only in pipeline.py
• Added docs in pipeline.py noting domain-specific logic should live in examples

3. Increased code coverage
Added tests for components that weren't covered:

• tests/test_quality.py - 18 tests for DataQualityAssessor
• tests/test_plugins.py - 12 tests for PluginRegistry
• tests/test_examples.py - 8 tests for example functionality

All 69 tests passing (up from 32).

4. Started testing examples
Created test framework that verifies examples execute correctly, handle edge cases, and work with different output formats.

The codebase now follows a recipe/examples-based architecture where pipeline.py provides core utilities, plugins handle specific data sources, and examples demonstrate complete workflows. This aligns with our vision of plugins with recipes/examples.

feel free to request any changes or if you want this pushed!

[bellanov]

Way to get ahead on this. I'll take a look when I get a chance today.

[bellanov]

Is there a PR or a branch you can attach to the issue? I don't see one currently.

[aybanda]

@bellanov it is now available #86

[bellanov]

I hopped on the branch. Loving the work thus far.

I'll have some requests as I go along so we can harden this further.

Just be sure to git pull before you do any work, so you pick up whatever changes I may be making.

[bellanov]

After some planning, I've come to a few decisions, which I'll describe below and also update the tasks.

Upped the estimate for this to 20, which may increase further if the work is more intensive than expected.

Education

We'll be refactoring the project quite a bit, with the goal being to have a more educational approach. I believe I've mentioned the possibility of using these tools to teach students and developers alike, so we'll cater the project towards this endeavor.

This means updating documentation appropriately as we go along and achieving as close to 100% code coverage as possible.

Kaggle

We'll be creating various examples that exercise the existing plugins using tried and true datasets, obtained from Kaggle or other existing data sources. The primary focus will initially revolve around processing local data sets. For remote data sets, I'll mock that functionality using Docker. The tests targeting Docker will be marked as integration tests.

FantasyAce

I'll be refactoring things on the FantasyAce end, so the existing plugins will likely be deprecated. While diving into team versus player properties, I found it necessary to redo the entire data model, of which the existing functionality depends upon. Long story short, existing FantasyAce functionality will simply be purged and revisited as I rebuild the data model.

I'll update tasks accordingly.

[bellanov]

I'm starting by implementing some vary basic examples, starting by defining the quickstart.

It will just contain data about people and their address. There will be various scenarios (i.e., single file, multiple files), in order to test input consumption. We will assume and expect that all of the files will have the same schema.

Some of the other data sources I mentioned (i.e., Kaggle) contain huge data sources and will have some additional steps to get the data local. They've deployed their data sets as Python package, accessible via API key, which is inspring.

It seems to be standard to get the data locally, especially in Notebooks, before executing ETL. We will embrace this mentaility.

[bellanov]

I'll catch up the pipeline.py to the current state of things.

Upping this to 30, since we're somewhat going back to the drawing board.