🚀 Feat: seeders blocks + launch validation#27
Conversation
There was a problem hiding this comment.
Pull Request Overview
This pull request introduces a Markdown Multiplier Block that enables chunk-based processing of markdown documents, transforming single markdown files into multiple pipeline executions for each chunk. The key architectural change allows pipelines to have two execution modes: traditional single-input/single-output and multiplier fan-out mode.
Key Changes:
- Added
MarkdownMultiplierBlockusing LlamaIndex for markdown parsing and chunking - Introduced
BaseMultiplierBlockabstract class and dual-mode pipeline execution logic - Enhanced seed validation API endpoint with template variable extraction
- Updated Q&A Generation template to process markdown documents via chunking
- Modified frontend to support markdown file uploads and real-time seed validation
Reviewed Changes
Copilot reviewed 42 out of 42 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
lib/blocks/builtin/markdown_multiplier.py |
New multiplier block for markdown chunking with configurable parser types |
lib/blocks/base.py |
Added BaseMultiplierBlock base class for fan-out blocks |
lib/workflow.py |
Implemented dual execution modes with _execute_multiplier_pipeline method |
lib/job_processor.py |
Added conditional logic to handle multiplier vs normal pipeline execution |
app.py |
Added /api/seeds/validate endpoint and markdown file support in /generate |
models.py |
Added SeedValidationRequest model for validation API |
tests/blocks/test_markdown_multiplier.py |
Comprehensive test suite for new multiplier block |
tests/test_workflow.py |
Added tests for multiplier validation rules and execution |
tests/test_api.py |
Added extensive seed validation endpoint test coverage |
lib/templates/qa_generation.yaml |
Updated to use markdown multiplier with chunk-based generation |
lib/templates/__init__.py |
Enhanced template loader to support markdown seed files |
lib/blocks/builtin/text_generator.py |
Added get_required_fields method for template variable extraction |
lib/blocks/builtin/structured_generator.py |
Renamed prompt parameter to user_prompt with template variable extraction |
frontend/src/pages/Generator.tsx |
Added markdown file support, validation UI, and multiplier pipeline detection |
frontend/src/components/pipeline-editor/BlockPalette.tsx |
Updated to use backend-provided category field with new "seeders" category |
frontend/src/components/pipeline-editor/BlockNode.tsx |
Updated block categorization to use category field from block data |
frontend/src/pages/Pipelines.tsx |
Added markdown seed download support for templates |
pyproject.toml |
Added llama-index-core>=0.14.7 dependency |
docs/templates.md |
Updated Q&A Generation documentation to reflect markdown chunking |
.github/ISSUE_TEMPLATE/* |
Cleaned up comment formatting in issue templates |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| 2. | ||
| 3. |
There was a problem hiding this comment.
Missing step 1 in reproduction steps. The list should start with "1." before step 2.
| 2. | |
| 3. | |
| 1. | |
| 2. |
frontend/src/pages/Generator.tsx
Outdated
| } | ||
| }; | ||
| revalidate(); | ||
| }, [selectedPipeline]); |
There was a problem hiding this comment.
This effect re-runs whenever selectedPipeline changes and attempts to revalidate the file. However, it doesn't include file in the dependency array, which could lead to stale closure issues. The effect uses file but doesn't list it as a dependency. This is a React Hook dependency warning that should be addressed by either adding file to the dependencies or using a different approach.
| }, [selectedPipeline]); | |
| }, [selectedPipeline, file]); |
lib/blocks/builtin/text_generator.py
Outdated
| except Exception: | ||
| pass |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
| except Exception: | |
| pass | |
| except Exception as e: | |
| logger.warning(f"Failed to parse Jinja2 template for required fields: {e} (prompt: {prompt!r})") |
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 46 out of 46 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 57 out of 57 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
app.py:1
- Line 238:
seed.get('repetitions', 1)is called twice in the isinstance check. The second call should beseed.get('repetitions')without the default value of 1. Current code:isinstance(seed.get('repetitions', 1), int)will always return True because if 'repetitions' is missing, it defaults to 1 (an int), masking potential type errors. Should be:isinstance(seed.get('repetitions'), int) if 'repetitions' in seed else True
import json
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Description
Find it in the related issue. Key changes:
Related Issue
Closes #26
Checklist
make formatpassesmake pre-mergepasses