Release 2.8.0

What's new

New Features

• Add an environment variable to control invalid document evaluation behavior
• Add WDOC_APPLY_ASYNCIO_PATCH env variable to manage asyncio patching
• Specify name of LocalFileStore for better logging
• Add a decorator for more useful debug logs

Improvements

Logging

• Introduced better format for debug prints
• Harmonized default environment value presentation in the documentation

Refactoring

• Replaced hash-based source identifiers with a consistent format
• Improved source identifier handling for single document cases

Bug Fixes

• MAJOR Resolved error where sources were not properly referenced
• MAJOR Addressed problems with cluster detection in text analysis
• Applied patch before running tests to resolve buggy processes
• Corrected issues with concurrency setting causing unexpected behavior
• Fixed an obsolete script referencing an outdated environment variable

Documentation

• Removed outdated mention of winston doc, replacing with current references

Dependency Management

• To run tests, pytest-xdist must be installed
• Bumped PersistDict to the latest version

Minor Changes

• Various minor code and logic corrections throughout the codebase

Commits details since the last release

• [bf143b9] by @thiswillbeyourgithub, 16 minutes ago:
  bump version 2.7.1 -> 2.8.0

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

• [f97935e] by @thiswillbeyourgithub, 3 minutes ago:
  Revert "tests: remove the fixture from tests as they are bugging some tests"
  This reverts commit 34adb42.

tests/conftest.py

• [6515373] by @thiswillbeyourgithub, 6 minutes ago:
  fix: apply patch before running tests
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/conftest.py
tests/run_all_tests.sh

• [34adb42] by @thiswillbeyourgithub, 14 minutes ago:
  tests: remove the fixture from tests as they are bugging some tests
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/conftest.py

• [10633a6] by @thiswillbeyourgithub, 61 minutes ago:
  fix: cant use xdist for the api tests apparently
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/run_all_tests.sh

• [5e7233b] by @thiswillbeyourgithub, 77 minutes ago:
  fix: obsolete script was using an old import env var name
  Signed-off-by: thiswillbeyourgithub [email protected]

scripts/AnkiFiltered/AnkiFilteredDeckCreator.py

• [7265e00] by @thiswillbeyourgithub, 2 hours ago:
  fix: to run the tests we must install pytest-xdist
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/run_all_tests.sh

• [ed11934] by @thiswillbeyourgithub, 2 hours ago:
  fix: to run the tests we need to patch asyncio
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/run_all_tests.sh

• [f382e17] by @thiswillbeyourgithub, 4 hours ago:
  new: better format for debug prints
  Signed-off-by: thiswillbeyourgithub
  [email protected]

wdoc/utils/logger.py

• [e213351] by @thiswillbeyourgithub, 4 hours ago:
  minor: remove a mention of winston doc and replace by wdoc
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md

• [11b5c95] by @thiswillbeyourgithub, 4 hours ago:
  fix: set default concurrency to 1 actually because it is causing issues
  Signed-off-by: thiswillbeyourgithub
  [email protected]

wdoc/docs/help.md
wdoc/utils/env.py

• [dca2b47] by @thiswillbeyourgithub (aider), 7 hours ago:
  feat: Add environment variable to control invalid document evaluation behavior

wdoc/docs/help.md
wdoc/utils/env.py
wdoc/utils/tasks/query.py

• [ffa2d67] by @thiswillbeyourgithub, 7 hours ago:
  docs: harmonize default env valuee presentation
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md

• [95ec3aa] by @thiswillbeyourgithub, 7 hours ago:
  fix: set default llm concurrency to 5 instead of 15
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md
wdoc/utils/env.py

• [c056ba4] by @thiswillbeyourgithub, 7 hours ago:
  fix: exit code should have been 0 not 1
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [26ba02b] by @thiswillbeyourgithub, 7 hours ago:
  fix: litellm debugging
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [9ac4f32] by @thiswillbeyourgithub, 7 hours ago:
  typo
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/tasks/query.py

• [3d3006e] by @thiswillbeyourgithub (aider), 31 hours ago:
  feat: Add WDOC_APPLY_ASYNCIO_PATCH env variable to control asyncio patching

wdoc/init.py
wdoc/docs/help.md
wdoc/utils/env.py

• [dfccef0] by @thiswillbeyourgithub, 31 hours ago:
  minor
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/init.py

• [ed5590b] by @thiswillbeyourgithub, 32 hours ago:
  new: reworked how we parse the outputs of eval_llm
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/tasks/query.py

• [f8cd452] by @thiswillbeyourgithub, 32 hours ago:
  new: use more debug logging
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/embeddings.py
wdoc/utils/llm.py
wdoc/utils/tasks/query.py
wdoc/wdoc.py

• [796f8cf] by @thiswillbeyourgithub, 32 hours ago:
  major fix: finally fix the bug where some sources were not properly referenced
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py
wdoc/utils/prompts.py
wdoc/utils/tasks/query.py
wdoc/wdoc.py

• [6538910] by @thiswillbeyourgithub, 33 hours ago:
  new: actually no need to use a debug print for the md printing
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py

• [3201936] by @thiswillbeyourgithub, 34 hours ago:
  fix: do not use info level debug print for the final answer of query
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py

• [357925f] by @thiswillbeyourgithub, 34 hours ago:
  fix: in rare cases where the clustering only finds 1 cluster

wdoc/utils/tasks/query.py

• [072c00d] by @thiswillbeyourgithub, 34 hours ago:
  new: allow finding oonly 2 clusters in a group of texts
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/tasks/query.py

• [1786ff7] by @thiswillbeyourgithub, 34 hours ago:
  fix: avoid issues when asking for more clusters than texts
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/tasks/query.py

• [8238599] by @thiswillbeyourgithub, 35 hours ago:
  bump persistdict
  Signed-off-by: thiswillbeyourgithub [email protected]

setup.py

• [fd01302] by @thiswillbeyourgithub, 2 days ago:
  new: add a decorator to get more useful debug logs
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py
wdoc/utils/tasks/query.py
wdoc/utils/tasks/summarize.py

• [4b58b16] by @thiswillbeyourgithub, 2 days ago:
  add more debug logs
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [971cda5] by @thiswillbeyourgithub, 2 days ago:
  apply black
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [639e443] by @thiswillbeyourgithub, 2 days ago:
  fix: use the up to date way to increase litellm log level

wdoc/wdoc.py

• [ef7cb6b] by @thiswillbeyourgithub, 2 days ago:
  new: add more debug prints
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/embeddings.py

• [dc5d203] by @thiswillbeyourgithub, 2 days ago:
  remove unused status function
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/embeddings.py

• [d836e6a] by @thiswillbeyourgithub, 2 days ago:
  new: set USER_AGENT before importing wdoc and its dep

wdoc/init.py
wdoc/wdoc.py

• [5b24a9f] by @thiswillbeyourgithub, 2 days ago:
  bump persistdict
  Signed-off-by: thiswillbeyourgithub [email protected]

setup.py

• [55ac32b] by @thiswillbeyourgithub, 2 days ago:
  update gitignore
  Signed-off-by: thiswillbeyourgithub
  [email protected]

.gitignore

• [4a75037] by @thiswillbeyourgithub, 2 days ago:
  new: specify name of LocalFileStore
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/customs/fix_llm_caching.py
wdoc/utils/embeddings.py
wdoc/utils/retrievers.py

• [55d0396] by @thiswillbeyourgithub, 2 days ago:
  new: use my custom LocalFileStore for retrievers too
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/retrievers.py

• [8b5269d] by @thiswillbeyourgithub, 2 days ago:
  bump PersistDict
  Signed-off-by: thiswillbeyourgithub [email protected]

setup.py

• [84850f7] by @thiswillbeyourgithub, 2 days ago:
  new: add a name to the LocalFileStore for better logging
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/customs/compressed_embeddings_cacher.py

• [c379df4] by @thiswillbeyourgithub, 2 days ago:
  fix: typo
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/customs/compressed_embeddings_cacher.py

• [cbf1c54] by @thiswillbeyourgithub (aider), 2 days ago:
  refactor: Replace hash-based source identifiers with consistent [[WDOC_N]] format

wdoc/utils/tasks/query.py
wdoc/wdoc.py

• [188eb64] by @thiswillbeyourgithub (aider), 2 days ago:
  fix: Correct source substitution logic for single document case

wdoc/wdoc.py

• [7fd2879] by @thiswillbeyourgithub (aider), 2 days ago:
  refactor: Improve source identifier handling for single document case

wdoc/wdoc.py

• [ef53691] by @thiswillbeyourgithub (aider), 2 days ago:
  fix: Resolve source substitution issue for single document case

wdoc/wdoc.py

• [656094c] by @thiswillbeyourgithub (aider), 2 days ago:
  fix: Improve source substitution and remove duplicate code block

wdoc/wdoc.py

• [f7ae6a3] by @thiswillbeyourgithub (aider), 2 days ago:
  fix: Ensure source_hashes is populated for single-document queries

wdoc/wdoc.py

• [bb3f8a3] by @thiswillbeyourgithub, 2 days ago:
  fix: wrong type hint
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [b89ff57] by @thiswillbeyourgithub (aider), 3 days ago:
  fix: Correct source substitution logic in query_task method

wdoc/wdoc.py

• [91c6f33] by @thiswillbeyourgithub, 3 days ago:
  Revert "refactor: Improve source hash tracking and batching in intermediate answer processing"
  This reverts commit 06501a3.

wdoc/wdoc.py

• [06501a3] by @thiswillbeyourgithub (aider), 3 days ago:
  refactor: Improve source hash tracking and batching in intermediate answer processing

wdoc/wdoc.py