Improve Testing Infrastructure for Codegen SDK #125
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Motivation The **Codegen on OSS** package provides a pipeline that: - **Collects repository URLs** from different sources (e.g., CSV files or GitHub searches). - **Parses repositories** using the codegen tool. - **Profiles performance** and logs metrics for each parsing run. - **Logs errors** to help pinpoint parsing failures or performance bottlenecks. <!-- Why is this change necessary? --> # Content <!-- Please include a summary of the change --> see [codegen-on-oss/README.md](https://github.com/codegen-sh/codegen-sdk/blob/acfe3dc07b65670af33b977fa1e7bc8627fd714e/codegen-on-oss/README.md) # Testing <!-- How was the change tested? --> `uv run modal run modal_run.py` No unit tests yet 😿 # Please check the following before marking your PR as ready for review - [ ] I have added tests for my changes - [x] I have updated the documentation or added new documentation as needed
Original commit by Tawsif Kamal: Revert "Revert "Adding Schema for Tool Outputs"" (codegen-sh#894) Reverts codegen-sh#892 --------- Co-authored-by: Rushil Patel <[email protected]> Co-authored-by: rushilpatel0 <[email protected]>
Original commit by Ellen Agarwal: fix: Workaround for relace not adding newlines (codegen-sh#907)
Reviewer's GuideThis PR overhauls the Codegen SDK’s testing infrastructure by updating pytest settings and coverage reporting, reorganizing and documenting test suites, introducing helper scripts for coverage and flaky test analysis, providing sample tests for key modules, and integrating automated CI workflows. Diagram for New Testing Helper ScriptsclassDiagram
class generate_coverage_badge_py {
<<Script>>
+parse_args()
+get_coverage_percentage()
+generate_badge(coverage_percentage, output_file)
+main()
}
note for generate_coverage_badge_py "Generates a test coverage SVG badge."
class analyze_test_coverage_py {
<<Script>>
+parse_args() (expected)
+analyze_coverage() (expected)
+generate_report() (expected)
+main() (expected)
}
note for analyze_test_coverage_py "Analyzes coverage to find low-coverage modules."
class find_flaky_tests_py {
<<Script>>
+parse_args() (expected)
+run_tests_repeatedly() (expected)
+identify_flaky_tests() (expected)
+main() (expected)
}
note for find_flaky_tests_py "Identifies flaky tests by repeated execution."
class run_tests_with_coverage_sh {
<<Shell Script>>
+ Executes pytest with coverage
+ Generates text and HTML reports
+ Supports parallel execution options
}
note for run_tests_with_coverage_sh "Orchestrates test execution and coverage reporting."
Class Diagram for New and Reorganized Test ModulesclassDiagram
class AbstractTest {
<<Abstact>>
#setup_method()
#teardown_method()
}
class TestTracer {
<<TestClass>>
+test_tracer_init()
+test_tracer_span_creation()
# ... other specific test methods for tracer.py
}
note for TestTracer "New unit tests for `agents/tracer.py`"
class TestUtils {
<<TestClass>>
+test_util_function_A()
+test_util_function_B_with_param(param)
# ... other specific test methods for utils.py
}
note for TestUtils "New unit tests for `agents/utils.py`"
class TestEvents {
<<TestClass>>
+test_event_processing()
+test_event_integration_with_X()
# ... other specific test methods for events.py
}
note for TestEvents "Integration tests for `codegen/extensions/events/test_events.py` (moved/updated)"
TestTracer --|> AbstractTest
TestUtils --|> AbstractTest
TestEvents --|> AbstractTest
class TestTemplate_py {
<<Test File Template>>
+ placeholder_test_function()
+ example_fixture_usage()
}
note for TestTemplate_py "`tests/templates/test_template.py` guides new test creation"
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Important Review skippedBot user detected. To trigger a single review, invoke the You can disable this status message by setting the 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Join our Discord community for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
Author
|
I see a check failed - I'm on it! 🫡 |
Author
|
✅ Fixed the failing checks in this commit for PR #125. The issues were:
The changes have been pushed to the original PR branch. 💻 View my work • React 👍 or 👎 |
Author
|
I see a check failed - I'm on it! 🫡 |
…mplify tests workflow
Author
|
✅ Fixed the failing checks in this commit for PR #125. The issues were:
These changes should resolve the failing pre-commit, mypy, and tests checks. The changes have been pushed to the original PR branch. 💻 View my work • React 👍 or 👎 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR improves the testing infrastructure for the Codegen SDK to address the issues identified in ZAM-415.
Changes Made
Fixed Test Configuration
Improved Test Organization
Enhanced Test Documentation
Added Testing Tools
Added Sample Tests
Added CI Integration
Benefits
Related Issues
💻 View my work • About Codegen
Summary by Sourcery
Overhaul the Codegen SDK testing infrastructure by standardizing configurations, adding test tooling and templates, organizing and documenting tests, and integrating automated CI testing
Enhancements:
Build:
CI:
Documentation:
Tests: