Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
105 commits
Select commit Hold shift + click to select a range
3d324ad
refactor: Use constants correctly in the src/pytest_brightest/plugin.…
gkapfham Jul 15, 2025
8726747
refactor: Use two more constants correctly in the src/pytest_brightes…
gkapfham Jul 15, 2025
ea4c277
docs: Add more details about the new logging refactoring to the docs/…
gkapfham Jul 16, 2025
bbd47ca
docs: Add the ARGUMENTS.md file to outline some examples of commands …
gkapfham Jul 16, 2025
b19deff
fix: Correct the numbering issue in the docs/plan.md file.
gkapfham Jul 16, 2025
1996090
docs: Add a comprehensive example of what the JSON file should look l…
gkapfham Jul 16, 2025
9c24a36
refactor: Improve the approach to saving execution data across multip…
gkapfham Jul 16, 2025
4900784
refactor: Improve the saving of data in the JSON report file by updat…
gkapfham Jul 16, 2025
423f74f
style: Improve comments and spacing and resolve minor linting issues …
gkapfham Jul 16, 2025
21aef4a
refactor: Comment out some not-needed constants and then improve src/…
gkapfham Jul 16, 2025
f0310e8
refactor: Delete the two comment-out constants in the src/pytest_brig…
gkapfham Jul 16, 2025
bfdcdfe
fix: Improve the saving of data in the pytest_brightest/plugin.py file.
gkapfham Jul 16, 2025
e84b17d
refactor: Improve the saving of data and the test suite.
gkapfham Jul 16, 2025
7e773dc
fix: Correct the name of a key for the data dictionary of saved data …
gkapfham Jul 16, 2025
0367df0
refactor: Comment out many unused constants in src/pytest_brightest/c…
gkapfham Jul 16, 2025
ed876a2
refactor: Remove a commented-out constant in the src/pytest_brightest…
gkapfham Jul 16, 2025
d96cbdb
refactor: Delete a large block of commented-out constants in src/pyte…
gkapfham Jul 16, 2025
75cdfb2
docs: Add the badge and link for the download count to the README.md …
gkapfham Jul 17, 2025
2d4e626
coms: Remove commented-out method for sorting of keys inside of src/p…
gkapfham Jul 21, 2025
23d6dc3
refactor: Change command-line argument name and improve comments in s…
gkapfham Jul 21, 2025
310d7d5
refactor: Add a new constant for TESTS_WITHIN_SUITE in src/pytest_bri…
gkapfham Jul 21, 2025
503d85a
coms: Add comments to explain setup steps for shuffling technique in …
gkapfham Jul 21, 2025
fa096ba
refactor: Start to use TESTS_WITHIN_SUITE as the new constant in src/…
gkapfham Jul 21, 2025
44b6921
docs: Add the additional notes section to the end of the GEMINI.md file.
gkapfham Jul 21, 2025
016033e
docs: Add the current plan for logging to the docs/plan.md file.
gkapfham Jul 21, 2025
22a3732
docs: Change the section to current plan in docs/plan.md.
gkapfham Jul 21, 2025
19da102
docs: Add more details about notetaking for the plan in the GEMINI.md…
gkapfham Jul 21, 2025
423dc2a
refactor: Fix test suite in tests/test_reorder.py so that it uses tes…
gkapfham Jul 21, 2025
fc29d4e
refactor: Improve the src/pytest_brightest/plugin.py file so that it …
gkapfham Jul 21, 2025
6007005
docs: Add more details about notes and planning to the docs/plan.md f…
gkapfham Jul 21, 2025
88747a2
docs: Change the phrase at the bottom of the GEMINI.md file.
gkapfham Jul 21, 2025
64b648d
docs: Add the additional notes section to the AGENTS.md file.
gkapfham Jul 21, 2025
14d606c
docs: Improve the content about additional notes in the GEMINI.md file.
gkapfham Jul 21, 2025
41f5127
refactor: Use constants and improve comments in the src/pytest_bright…
gkapfham Jul 21, 2025
b34f4ab
refactor: Remove a commented-out constant in the src/pytest_brightest…
gkapfham Jul 21, 2025
d280975
coms: Delete two commented-out constants in the src/pytest_brightest/…
gkapfham Jul 21, 2025
c3d6e8f
refactor: Reduce the number of spaces inside of INDENT to be two fewe…
gkapfham Jul 21, 2025
f6fe436
feat: Add diagnostic output for checking the cost of tests in the src…
gkapfham Jul 21, 2025
fbff1d1
refactor: Remove duplicative console output for test costs in the src…
gkapfham Jul 21, 2025
e31047b
feat: Add diagnostic messages to the console about the reordering acc…
gkapfham Jul 21, 2025
016bfae
coms: Add a further comment to the src/pytest_brightest/reorder.py file.
gkapfham Jul 21, 2025
469bc53
refactor: Start to improve the reordering of tests within module for …
gkapfham Jul 21, 2025
ddb0456
test: Add, for now, a commented-out test case in the tests/test_reord…
gkapfham Jul 21, 2025
8522000
fix: Correct the feature that did not handle test failure correctly f…
gkapfham Jul 21, 2025
f195722
test: Add more test cases for the improvement of reordering feature i…
gkapfham Jul 21, 2025
bbf2fe8
fix: Correct the way that tests are reordered by failure in the src/p…
gkapfham Jul 21, 2025
f28ef11
test: Add and update test cases that are focused on test failure reor…
gkapfham Jul 21, 2025
163a455
fix: Clarify the console logging output for reordering in the src/pyt…
gkapfham Jul 21, 2025
bd68cbc
refactor: Make sure that data is saved in an un-accounted for config …
gkapfham Jul 21, 2025
e4e0d07
test: Add more assertions to a test case in tests/test_reorder.py.
gkapfham Jul 21, 2025
63a6e2b
refactor: Correct a console logging message and then improve messages…
gkapfham Jul 21, 2025
68a8fc4
docs: Update the docs/plan.md file with details about the --repeat fe…
gkapfham Jul 21, 2025
9b6235a
feat: Add an implementation of --repeat to the src/pytest_brightest/p…
gkapfham Jul 21, 2025
39251a3
test: Add more test cases to handle the --repeat functionality in tes…
gkapfham Jul 21, 2025
d3280f8
docs: Add more details to the docs/plan.md file about the --repeat an…
gkapfham Jul 21, 2025
d2ed5e9
chore: Add a commented-out feature that does not work yet in the src/…
gkapfham Jul 21, 2025
9f59329
test: Add more assertions and tests to support new feature for repeat…
gkapfham Jul 21, 2025
569b10f
feat: Add first implementation of the --repeat-failed feature in src/…
gkapfham Jul 21, 2025
ddc4e27
refactor: Improve and correct aspects of --repeat and --repeat-failed…
gkapfham Jul 21, 2025
263375d
refactor: Change the order of output in the src/pytest_brightest/plug…
gkapfham Jul 21, 2025
8c96b37
test: Fix the incorrect assertions corresponding to the re-arranged o…
gkapfham Jul 21, 2025
796d02f
feat: Add more diagnostic information about test case repeating to sr…
gkapfham Jul 21, 2025
097da30
docs: Modify a subsection title in the docs/plan.md file.
gkapfham Jul 22, 2025
c7ba756
test: Add a test case about the final run's performance data in tests…
gkapfham Jul 22, 2025
3907199
test: Improve the default values for testing the plugin in the tests/…
gkapfham Jul 22, 2025
8f25eb8
test: Update the string that is checked for in the tests/test_plugin.…
gkapfham Jul 22, 2025
527b9f1
feat: Improve diagnostic output for two repeat features in src/pytest…
gkapfham Jul 22, 2025
fbbacf7
refactor: Make the first attempt at the ratio calculation and use acr…
gkapfham Jul 22, 2025
49f9a02
refactor: Improve constants.py and plugin.py so that it saves data in…
gkapfham Jul 22, 2025
e9db527
refactor: Improve further the content in reorder.py and test_reorder.…
gkapfham Jul 22, 2025
33a415e
refactor: Further improve the calculation and storage of the ratio da…
gkapfham Jul 22, 2025
5037b4d
refactor: First attempt at adding the tie-breaking approach across mu…
gkapfham Jul 22, 2025
e75d5c8
refactor: Add an improved implementation for inverse-cost and inverse…
gkapfham Jul 22, 2025
85a8276
style: Improve the formatting for the options for the tiebreaker in s…
gkapfham Jul 22, 2025
a617c0a
docs: Add details about project requirements to the AGENTS.md file.
gkapfham Jul 22, 2025
4e6552a
docs: Add details about the project requirements and workflow to the …
gkapfham Jul 22, 2025
c8cc094
refactor: Make tie-breaker work for a single option and improve overa…
gkapfham Jul 22, 2025
601114e
refactor: Remove not-needed constants for tie-breaking and otherwise …
gkapfham Jul 22, 2025
b432feb
refactor: Add a complete (and partially tested) tie-breaking implemen…
gkapfham Jul 22, 2025
56eff45
coms: Slightly improve the comments in test_reorder.py even though th…
gkapfham Jul 22, 2025
33d293e
docs: Add the reminder about test case output to the GEMINI.md file.
gkapfham Jul 22, 2025
ebb8498
docs: Add the reminder about test case output to the AGENTS.md file.
gkapfham Jul 22, 2025
a620c96
fix: Make many enhancements to ensure that tie-breaking works in all …
gkapfham Jul 22, 2025
922dc5f
test: Improve mocking in tests so that they do not produce console ou…
gkapfham Jul 22, 2025
e26fbda
docs: Add a rough, tentative plan for empirical evaluation and improv…
gkapfham Jul 22, 2025
0e9f52d
test: Add more test cases to the tests/test_shuffler.py file.
gkapfham Jul 22, 2025
7d568ed
docs: Update parts of a plan that were finished in the docs/plan.md f…
gkapfham Jul 22, 2025
26c42dc
refactor: Remove one of the not-needed NODEID constants in in src/pyt…
gkapfham Jul 22, 2025
4b502cc
refactor: Correctly use self in a console.print call in src/pytest_br…
gkapfham Jul 22, 2025
216abdd
docs: Fix formatting and slightly improve phrasing in the docs/plan.m…
gkapfham Jul 22, 2025
93915d9
docs: Add more details to the "novel" experimental plan, he he, in do…
gkapfham Jul 22, 2025
fa66570
feat: Add inverse-name tie-breaker option to complement existing inve…
gkapfham Jul 22, 2025
14155f3
fix: Add use of INVERSE_NAME and make it a viable argument input in s…
gkapfham Jul 22, 2025
77b025d
coms: Add more comments to explain the JSON file in src/pytest_bright…
gkapfham Jul 22, 2025
ccb5968
refactor: Add first pass at a strategy for overriding the max test ru…
gkapfham Jul 22, 2025
08cabde
test: Refactor the entire tests/test_plugin.py test suite to avoid a …
gkapfham Jul 22, 2025
bdfcd48
refactor: Change slightly command-line argument descriptions in src/p…
gkapfham Jul 22, 2025
34c8b0c
docs: Add some DONE markers to subsections in the docs/plan.md file.
gkapfham Jul 22, 2025
816cb4f
docs: Add two more "for later" reminders in the docs/plan.md file.
gkapfham Jul 22, 2025
def48d4
docs: Add starting version of the description for the historical data…
gkapfham Jul 22, 2025
c6368e1
chore: Try to add the historical data feature across multiple files.
gkapfham Jul 22, 2025
1967f48
refactor: Ensure that average-* features are being called correctly a…
gkapfham Jul 22, 2025
b21856b
refactor: Improve display of diagnostic output about reordering in sr…
gkapfham Jul 22, 2025
e47afb0
fix: Remove a not-needed console.print statement in the src/pytest_br…
gkapfham Jul 22, 2025
729674c
test: Use parameters with _ assignment for not-needed, but signature-…
gkapfham Jul 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,26 @@ imports into the middle of a file or even at the start of a function or class.
- Support Python 3.11, 3.12, and 3.13 on MacOS, Linux, and Windows.
- Use Pydantic models for data validation and JSON serialization.

## Project Behavior Requirements

- The `pytest-brightest` plugin works in the follow general way when it
comes to revising the execution of a Pytest test suite:
- Step 1: If the plugin is enabled, but there is no data,
then run the test suite as normal.
- Step 2: If the plugin is enabled and there is data, then run the
test suite based on the data from the prior run of the plugin.
- Step 3: Use the data collected by pytest-json-report to compute
the values for the cost and failure data for each test case and
for the test modules.
- Step 4: While keeping all prior data that was recorded by the tool,
the plugin will then update the data with the new cost and failure
data for each test case and for the test modules.
- Step 5: The plugin will then write the updated data to the
`pytest-brightest.json` file in the configured directory.
- Critically, the `pytest-brightest` plugin should not, for instance,
delete data from a prior run unless the tool is beyond the maximum
amount of test runs that it is configured to persist.

## Test Requirements

All test cases should follow these standards:
Expand All @@ -94,6 +114,7 @@ All test cases should follow these standards:
- Test cases should aim to achieve full function, statement, and branch
coverage.
- Property-based tests must be marked with `@pytest.mark.property`.
- Test cases should not produce any console output.

## Making Changes

Expand All @@ -110,3 +131,11 @@ All test cases should follow these standards:
7. **Completion**: When you are finished with tasks, please summarize what tasks
you completed, how you completed them, the challenges you faced, how you
overcame them, and the rules that you followed during completion of the tasks.

## Additional Notes for the Coding Agent

Note: This file is specifically for coding agents and this section can be used
by the coding agent to add additional details about this project that would
be helpful for the agent to know when it is run again. Every time a coding
agent finishes a task, it should add notes here as it deems appropriate.
The coding agent should write the notes as a Markdown list.
62 changes: 62 additions & 0 deletions ARGUMENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Command-Line Arguments Checklist

This document provides a checklist of valid and invalid command-line arguments
for `pytest-brightest`. It is intended to be used as a guide for manual and
automated testing.

## Valid Command-Line Arguments

### Reordering by Cost

- [ ] `uv run pytest --json-report --brightest --reorder-by-technique cost --reorder-by-focus modules-within-suite --reorder-in-direction ascending`
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique cost --reorder-by-focus modules-within-suite --reorder-in-direction descending`
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique cost --reorder-by-focus tests-within-module --reorder-in-direction ascending`
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique cost --reorder-by-focus tests-within-module --reorder-in-direction descending`
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique cost --reorder-by-focus tests-across-modules --reorder-in-direction ascending`
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique cost --reorder-by-focus tests-across-modules --reorder-in-direction descending`

### Reordering by Name

- [ ] `uv run pytest --json-report --brightest --reorder-by-technique name --reorder-by-focus modules-within-suite --reorder-in-direction ascending`
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique name --reorder-by-focus modules-within-suite --reorder-in-direction descending`
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique name --reorder-by-focus tests-within-module --reorder-in-direction ascending`
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique name --reorder-by-focus tests-within-module --reorder-in-direction descending`
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique name --reorder-by-focus tests-across-modules --reorder-in-direction ascending`
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique name --reorder-by-focus tests-across-modules --reorder-in-direction descending`

### Reordering by Failure

- [ ] `uv run pytest --json-report --brightest --reorder-by-technique failure --reorder-by-focus modules-within-suite --reorder-in-direction ascending`
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique failure --reorder-by-focus modules-within-suite --reorder-in-direction descending`

### Shuffling

- [ ] `uv run pytest --json-report --brightest --reorder-by-technique shuffle --reorder-by-focus modules-within-suite`
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique shuffle --reorder-by-focus tests-within-module`
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique shuffle --reorder-by-focus tests-across-modules`
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique shuffle --seed 12345`

## Invalid Command-Line Arguments

### Missing `--brightest`

- [ ] `uv run pytest --json-report --reorder-by-technique cost` (Plugin should not activate)

### Invalid Technique

- [ ] `uv run pytest --json-report --brightest --reorder-by-technique invalid_technique` (Should raise an error)

### Invalid Focus

- [ ] `uv run pytest --json-report --brightest --reorder-by-technique cost --reorder-by-focus invalid_focus` (Should raise an error)

### Invalid Direction

- [ ] `uv run pytest --json-report --brightest --reorder-by-technique cost --reorder-in-direction invalid_direction` (Should raise an error)

### Conflicting Arguments

- [ ] `uv run pytest --json-report --brightest --reorder-by-technique shuffle --reorder-in-direction ascending` (Direction should be ignored)
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique cost --seed 12345` (Seed should be ignored)
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique failure --reorder-by-focus tests-within-module` (Not a valid combination)
- [ ] `uv run pytest --json-report --brightest --reorder-by-technique failure --reorder-by-focus tests-across-modules` (Not a valid combination)
55 changes: 44 additions & 11 deletions GEMINI.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,20 +7,24 @@ changes to this repository. Your primary goal is to make safe, incremental,
and high-quality contributions.

- **Always use `uv`:** This project uses `uv` for all dependency management,
virtual environments, and task running. Do not use `pip` or `venv` directly.
virtual environments, and task running. Do not use `pip` or `venv` directly.
- **Follow all guidelines:** This document contains the complete set of
guidelines from `AGENTS.md` and `docs/plan.md`. You must follow them strictly.
guidelines from `AGENTS.md` and `docs/plan.md`. You must follow them strictly.
- **Verify your changes:** Before committing any changes, you must run all
linters and tests to ensure your changes are correct and follow the project's
style. Use `uv run task all`.
- **Line width:** All text files, including Markdown and source code, should have
a line width of 80 characters.
- **Permission to run commands:** You have permission to run all commands in this
file to verify their functionality.
- **Incremental changes:** Make small, incremental changes. This makes it easier
to review your work and catch errors early.
linters and tests to ensure your changes are correct and follow the project's
style. Use `uv run task all`.
- **Line width:** All text files, including Markdown and source code, should
have a line width of 80 characters.
- **Permission to run commands:** You have permission to run all commands in
this file to verify their functionality.
- **Incremental changes:** Make small, incremental changes. This makes it
easier to review your work and catch errors early.
- **Communicate clearly:** When you propose changes, explain what you've done
and why.
and why.
- **Save your plan:** Save a copy of your plan for completing a task in the
`docs/plan.md` file. There is a section called `Current Plan` at the bottom of
the file and you can save your plan there. Create a new sub-sub-section with
the title of your plan and the date you plan to complete it.

As a Gemini agent, you must also follow these behavior guidelines, especially
when it comes to notifying the programmer about your work and status:
Expand Down Expand Up @@ -83,6 +87,26 @@ messages.
- Support Python 3.11, 3.12, and 3.13 on MacOS, Linux, and Windows.
- Use Pydantic models for data validation and JSON serialization.

## Project Behavior Requirements

- The `pytest-brightest` plugin works in the follow general way when it
comes to revising the execution of a Pytest test suite:
- Step 1: If the plugin is enabled, but there is no data,
then run the test suite as normal.
- Step 2: If the plugin is enabled and there is data, then run the
test suite based on the data from the prior run of the plugin.
- Step 3: Use the data collected by pytest-json-report to compute
the values for the cost and failure data for each test case and
for the test modules.
- Step 4: While keeping all prior data that was recorded by the tool,
the plugin will then update the data with the new cost and failure
data for each test case and for the test modules.
- Step 5: The plugin will then write the updated data to the
`pytest-brightest.json` file in the configured directory.
- Critically, the `pytest-brightest` plugin should not, for instance,
delete data from a prior run unless the tool is beyond the maximum
amount of test runs that it is configured to persist.

## Test Requirements

All test cases should follow these standards:
Expand All @@ -98,6 +122,7 @@ All test cases should follow these standards:
- Test cases should aim to achieve full function, statement, and branch
coverage.
- Property-based tests must be marked with `@pytest.mark.property`.
- Test cases should not produce any console output.

## Making Changes

Expand All @@ -114,3 +139,11 @@ All test cases should follow these standards:
7. **Completion**: When you are finished with tasks, please summarize what tasks
you completed, how you completed them, the challenges you faced, how you
overcame them, and the rules that you followed during completion of the tasks.

## Additional Notes for the Gemini Agent

Note: This file is specifically for Gemini agents and this section can be used
by the Gemini agent to add additional details about this project that would
be helpful for the agent to know when it is run again. Every time a Gemini
agent finishes a task, it should add notes here as it deems appropriate.
The Gemini agent should write the notes as a Markdown list.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
[![Coverage](https://img.shields.io/badge/dynamic/json?color=brightgreen&label=coverage&query=%24.totals.percent_covered_display&suffix=%25&url=https%3A//raw.githubusercontent.com/wiki/AstuteSource/pytest-brightest/coverage.json)](https://github.com/AstuteSource/pytest-brightest)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pytest-brightest)
![PyPI - Version](https://img.shields.io/pypi/v/pytest-brightest?logoColor=blue&color=blue)
[![PyPI Downloads](https://static.pepy.tech/badge/pytest-brightest)](https://pepy.tech/projects/pytest-brightest)
[![Maintenance](https://img.shields.io/badge/maintained%3F-Yes-blue.svg)](https://github.com/AstuteSource/pytest-brightest/graphs/commit-activity)
[![License LGPL v3](https://img.shields.io/badge/license-LGPL%20v3-blue.svg)](https://www.gnu.org/licenses/lgpl-3.0)

Expand Down
Loading
Loading