Release 2.9.0

What's new

New Features

• Shell Command Testing
  • Added shell command test for NYTimes parsing with content validation. [d3832f2]

Fixes

• Intermediate Merging
  • Fixed error during batch merging, ensuring intermediate answers are handled properly when too large. [d06cbb3]
  • Improved logic for retry attempts during batch merging. [573e15f]
• Parsing Reliability
  • Prevent return of intermediately parsed output on parsing failure. [78c9364]
  • Adapted handling to retry when encountering unparseable output or finish_reason=length. [763e9b4]
• Backend and Output
  • Resolved backend errors and edge case issues breaking summary generation. [bf00ced, 89c01de]
  • Corrected handling of test outputs to avoid crashing in specific import modes. [ee416ec]
• Testing Corrections
  • Corrected various test functions and ensured expected outcomes align with API changes. [bfcba71, 89c01de, 292ce90]

Documentation

• General Updates
  • Enhanced walkthrough formatting for improved readability in documentation. [2ec7fad]
  • Provided context about the origins of wdoc in README. [31e6c5d]
• Example and Help Docs
  • Updated examples documentation to remove confusing import_mode arguments. [cef5cdf]
  • Improved query and summary help documentation to reflect recent changes. [a289759, f1a6294]

Improvements

• Configuration and Setup Adjustments
  • Simplified the post-install script to handle dependencies via uv if present. [7a55143, 4ed0a35]
• Performance and Debugging Enhancements
  • Bumped max_tokens for intermediate answers to accommodate larger outputs during processing. [df059b6]
  • Stored original strings before parsing for effective debugging. [6ec3957]
  • Addressed deprecation warnings to keep up with latest standards. [15e4793]

Minor Changes

• Code and Debug Tune-ups
  • Streamlined testing arguments and outputs for precise validation. [7e5e4ce, b7f1a1f, d79b4c5]
  • Fine-tuned post-install scripts and functional debug outputs for better clarity. [d620f87]
• Enhanced wdoc Docs Via SVG Files (WIP)
  • Create SVG diagram and documentation for summary algorithm. [b5f49a9]
  • Added SVG visualization and improved design and intuitive flow representation for better understanding. [9489a8a, 703dde0]

Commits details since the last release

• [d06cbb3] by @thiswillbeyourgithub, 34 minutes ago:
  fix: error when merging batch when intermediate answers got so large the model can't merge them anymore
  We just concatennate them using semantic order and that will be good
  enough, the alternative is too expensive

Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [573e15f] by @thiswillbeyourgithub, 35 minutes ago:
  fix: one more trial given to merge batches
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [7edbe1f] by @thiswillbeyourgithub, 54 minutes ago:
  doc: add helpful debug message if abrupt message tail
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [df059b6] by @thiswillbeyourgithub, 55 minutes ago:
  new: bump max_token for intermediate answer from 1000 to 4000
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [78c9364] by @thiswillbeyourgithub, 3 hours ago:
  fix: don't return intermediately parsed output if parsing fails
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [6ec3957] by @thiswillbeyourgithub, 3 hours ago:
  minor: store the original string before parsing to help debugging
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [ee1c857] by @thiswillbeyourgithub, 3 hours ago:
  minor: better order of the output price prints
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [cc49037] by @thiswillbeyourgithub, 3 hours ago:
  fix: out_file test
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [cef5cdf] by @thiswillbeyourgithub, 3 hours ago:
  fix: forgot to remove import_mode args from examples
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/examples.md

• [d3832f2] by @thiswillbeyourgithub (aider), 3 hours ago:
  feat: Add shell command test for NYTimes parsing with content validation

tests/test_wdoc.py

• [ee416ec] by @thiswillbeyourgithub, 3 hours ago:
  new: don't crash if using import_mode at the same time as out_file
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [32ecdbd] by @thiswillbeyourgithub, 4 hours ago:
  test: remove unused debug and verbose argsc
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [a13d20a] by @thiswillbeyourgithub, 4 hours ago:
  new: remove confusing arg 'import_mode' and set it automatically depending on if imported or launched from cli
  Signed-off-by: thiswillbeyourgithub
  [email protected]

README.md
scripts/AnkiFiltered/AnkiFilteredDeckCreator.py
scripts/TheFiche/TheFiche.py
tests/test_wdoc.py
wdoc/main.py
wdoc/docs/help.md
wdoc/wdoc.py

• [763e9b4] by @thiswillbeyourgithub, 4 hours ago:
  fix: now if eval_llm returns something unparsable or with finish_reason=length we always retry
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [15e4793] by @thiswillbeyourgithub, 4 hours ago:
  minor: address deprecation warnings
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/tasks/query.py

• [b7f1a1f] by @thiswillbeyourgithub, 4 hours ago:
  test: set semantic batching test to api mark
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [7e5e4ce] by @thiswillbeyourgithub, 4 hours ago:
  test: move semantic batching test to the api section
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [e5530b6] by @thiswillbeyourgithub, 5 hours ago:
  test: add test for mistral embeddings
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [d79b4c5] by @thiswillbeyourgithub, 5 hours ago:
  fix: remove unused arg in tests
  Signed-off-by: thiswillbeyourgithub
  [email protected]
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [13ad2aa] by @thiswillbeyourgithub, 5 hours ago:
  test: ollama should be an api mark not basic
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [d3679b3] by @thiswillbeyourgithub, 5 hours ago:
  minor: sort pytest by mark
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [bfcba71] by @thiswillbeyourgithub, 6 hours ago:
  fix test
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [292ce90] by @thiswillbeyourgithub, 6 hours ago:
  fix: test of query
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [740dc25] by @thiswillbeyourgithub, 6 hours ago:
  fix: test of out_file
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [ff73711] by @thiswillbeyourgithub, 6 hours ago:
  fix: summary test
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [3b292ea] by @thiswillbeyourgithub, 6 hours ago:
  fix: remove unused arg in tests
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [bf00ced] by @thiswillbeyourgithub, 6 hours ago:
  fix: edge case was breaking summary
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [89c01de] by @thiswillbeyourgithub, 6 hours ago:
  fix: backend error in one edge case
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [ebcf92f] by @thiswillbeyourgithub (aider), 6 hours ago:
  feat: Change default query relevancy threshold to -0.5

wdoc/docs/help.md
wdoc/wdoc.py

• [443aab4] by @thiswillbeyourgithub, 7 hours ago:
  fix: query_task arg is actually optional
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [5847cd3] by @thiswillbeyourgithub, 7 hours ago:
  fix: missing var if only one document present
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [31e6c5d] by @thiswillbeyourgithub (aider), 12 hours ago:
  docs: Add context about medical student's motivation for creating wdoc

README.md

• [2ec7fad] by @thiswillbeyourgithub (aider), 14 hours ago:
  style: Update walkthrough formatting to use triple backticks for code blocks

wdoc/docs/examples.md

• [c06a849] by @thiswillbeyourgithub, 15 hours ago:
  update roadmap
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [7d1ba8b] by @thiswillbeyourgithub, 15 hours ago:
  fix: link to examples
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [d003d8e] by @thiswillbeyourgithub, 25 hours ago:
  fix: ongoing fix for the summary test
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [3a5f805] by @thiswillbeyourgithub, 25 hours ago:
  fix: tests for api were wrong
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [17e6a0f] by @thiswillbeyourgithub, 26 hours ago:
  fix: test using out_file
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [50325e9] by @thiswillbeyourgithub, 26 hours ago:
  fix: dont read from stdin if pytest is imported
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [5337005] by @thiswillbeyourgithub, 26 hours ago:
  minor
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [14735c8] by @thiswillbeyourgithub, 26 hours ago:
  minor
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/batch_file_loader.py

• [4ed0a35] by @thiswillbeyourgithub, 26 hours ago:
  new: the post install script now tries to install python-magic from git
  Signed-off-by: thiswillbeyourgithub [email protected]

setup.py

• [7a55143] by @thiswillbeyourgithub, 26 hours ago:
  new: try to use uv for the PostInstall script
  Signed-off-by: thiswillbeyourgithub [email protected]

setup.py

• [7c70556] by @thiswillbeyourgithub, 26 hours ago:
  docs: minor
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [6d0c42c] by @thiswillbeyourgithub, 26 hours ago:
  feat: allow using shell pipes
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md
setup.py
wdoc/main.py
wdoc/docs/examples.md
wdoc/utils/misc.py

• [d620f87] by @thiswillbeyourgithub, 27 hours ago:
  docs: insist on txt vs text filetype
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md

• [a0c5495] by @thiswillbeyourgithub, 29 hours ago:
  minor
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/main.py

• [83df9c5] by @thiswillbeyourgithub, 30 hours ago:
  new: use Literal type hint for audio backend
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/loaders.py

• [0712804] by @thiswillbeyourgithub (aider), 2 days ago:
  test: Add API test for summary out_file argument

tests/test_wdoc.py

• [a289759] by @thiswillbeyourgithub (aider), 2 days ago:
  docs: Update documentation for out_file argument with query use case

wdoc/docs/help.md

• [f1a6294] by @thiswillbeyourgithub, 2 days ago:
  feat: allow out_file to be specified for query too
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [0b524a7] by @thiswillbeyourgithub, 2 days ago:
  harmonize the way out_file is handled
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py
wdoc/wdoc.py

• [d027430] by @thiswillbeyourgithub, 2 days ago:
  put unfinished svg into some folder
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/svg/query_animation.html
wdoc/docs/svg/query_rag.md
wdoc/docs/svg/summary.svg

• [44f36f0] by @thiswillbeyourgithub, 2 days ago:
  add
  Signed-off-by: thiswillbeyourgithub [email protected]

query_rag.md

• [4a673b0] by @thiswillbeyourgithub (aider), 2 days ago:
  refactor: Reposition step 4 and update process flow arrows in summary diagram

wdoc/docs/summary.svg

• [4ca2bdd] by @thiswillbeyourgithub (aider), 2 days ago:
  fix: Swap step 3 and 4 labels in summary SVG diagram

wdoc/docs/summary.svg

• [8703532] by @thiswillbeyourgithub, 2 days ago:
  better svg
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/summary.svg

• [b5f49a9] by @thiswillbeyourgithub (aider), 2 days ago:
  feat: Improve wdoc summary algorithm SVG diagram and documentation

summary_rag.md
wdoc/docs/summary.svg

• [204ca0f] by @thiswillbeyourgithub (aider), 2 days ago:
  refactor: Update SVG flow and remove distracting icons

wdoc/docs/summary.svg

• [703dde0] by @thiswillbeyourgithub (aider), 2 days ago:
  feat: Simplify SVG cycle with more intuitive circular design

wdoc/docs/summary.svg

• [fc83dd3] by @thiswillbeyourgithub (aider), 2 days ago:
  feat: Enhance SVG visualization with improved design, color scheme, and intuitive flow representation

wdoc/docs/summary.svg

• [9489a8a] by @thiswillbeyourgithub (aider), 2 days ago:
  feat: Add SVG visualization of wdoc summary algorithm

wdoc/docs/summary.svg

• [3e97575] by @thiswillbeyourgithub (aider), 2 days ago:
  fix: Encode '>' as HTML entity in query_animation.html

wdoc/docs/query_animation.html

• [37c8244] by @thiswillbeyourgithub (aider), 2 days ago:
  refactor: Convert dynamic SVG animation to static HTML flow diagram

wdoc/docs/query_animation.html

• [3746978] by @thiswillbeyourgithub, 2 days ago:
  remove unused arrows
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/query_animation.html

• [c9dd00c] by @thiswillbeyourgithub, 2 days ago:
  add query animation
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/query_animation.html