Release 3.2.3

What's new

This release primarily focuses on enhancing context management for embedding models, improving debugging utilities, and updating documentation for better clarity. It also includes several important bug fixes and feature additions.

✨ Features

• Introduced a new environment variable WDOC_MAX_EMBED_CONTEXT to allow capping the context size for embedding models ([d9e200f8])
  • Documentation for this new variable has been added ([a2408fd0])
• Enhanced debugging by ensuring debug prints are always active when md_printer is used. This helps in retrieving LLM answers from logs if they weren't saved to a file ([69db1916])
• Added the current date to summary metadata and headers to help reduce potential LLM hallucinations ([64ca4665])

🐛 Fixes

• Text Splitting & Context Handling:
  • Addressed an issue where large language models have more context than embedding models by setting a max_tokens limit for the text splitter ([dac6802d])
  • Fixed an edge case where the wdoc max chunk setting could be ignored ([196b3a00])
  • Corrected an old variable name within the text splitting logic ([767bc754])
• Updated the default model to gemini 2.5 preview to reflect its renaming on OpenRouter ([22978609])
• Improved the mechanism for ignoring initial "breathing" or placeholder lines in summaries ([4dbcf158])

📚 Documentation

• Clarity and Enhancements:
  • Clarified the usage of save and load functionalities ([9d9642d4]) and specifically advised against using them simultaneously ([5270c350])
  • Made multiple clarifications to the README for better understanding ([9284ff54], [cb4cb519], [f677e5a2], [39e0da55])
  • Updated Ollama examples to recommend snowflake-arctic-embed2 instead of bge-m3 ([d045702b])
  • Added documentation for the WDOC_MAX_EMBED_CONTEXT environment variable ([a2408fd0])
• Removed a documentation file (summary_rag.md) that was not yet ready for release ([6d20c220])

⚙️ Chore & Maintenance

• Version bumped to 3.2.3 (following an earlier bump to 3.2.2 [[71ac503c]]) ([f62a2322])
• README Updates:
  • Updated TODO items ([8f2cbfd7], [5d090421])
  • Added a PyPI badge for better project visibility ([60ef4112])

Commits details since the last release

• [f62a232] by @thiswillbeyourgithub, 46 seconds ago:
  bump version 3.2.2 -> 3.2.3

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

• [6d20c22] by @thiswillbeyourgithub, 76 seconds ago:
  doc: removed file not yet ready
  Signed-off-by: thiswillbeyourgithub [email protected]

summary_rag.md

• [71ac503] by @thiswillbeyourgithub, 4 minutes ago:
  bump version 3.2.1 -> 3.2.2

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

• [8f2cbfd] by @thiswillbeyourgithub, 3 minutes ago:
  todo
  Signed-off-by: thiswillbeyourgithub
  [email protected]

README.md

• [69db191] by @thiswillbeyourgithub, 40 minutes ago:
  new: now debug print is used anyway when md_printer is used
  this is to make you able to go to the logs to fetch and answer form the
  LLM if you have forgotten to store it to a file

Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py
wdoc/wdoc.py

• [a2408fd] by @thiswillbeyourgithub (aider), 66 minutes ago:
  docs: Add documentation for WDOC_MAX_EMBED_CONTEXT variable

wdoc/docs/help.md

• [d9e200f] by @thiswillbeyourgithub, 66 minutes ago:
  feat: add new env var to cap the context size for embedding models
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/env.py
wdoc/utils/misc.py

• [196b3a0] by @thiswillbeyourgithub, 72 minutes ago:
  fix: edge case where wdoc max chunk would be ignored
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [dac6802] by @thiswillbeyourgithub, 76 minutes ago:
  fix: set a limit to max_tokens for the text splitter as large LLM have more context than embeddings models nowadays
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [767bc75] by @thiswillbeyourgithub, 80 minutes ago:
  fix: forgot to rename an old variable name
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [2297860] by @thiswillbeyourgithub, 86 minutes ago:
  fix: set default model to gemini 2.5 preview without date timestamp
  openrouter renamed that model apparently

Signed-off-by: thiswillbeyourgithub [email protected]

README.md
wdoc/utils/env.py

• [9d9642d] by @thiswillbeyourgithub, 22 hours ago:
  doc: clarify save and load
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md

• [5270c35] by @thiswillbeyourgithub, 22 hours ago:
  doc: clarify that load and save shouldnt be used at the same time
  Signed-off-by: thiswillbeyourgithub
  [email protected]

wdoc/docs/help.md

• [d045702] by @thiswillbeyourgithub, 23 hours ago:
  doc: use snowflake-arctic-embed2 instead of bge-m3 for ollama examples
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/examples.md

• [60ef411] by @thiswillbeyourgithub, 26 hours ago:
  add a pypi badge
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [5d09042] by @thiswillbeyourgithub, 7 days ago:
  update todo
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [9284ff5] by @thiswillbeyourgithub, 7 days ago:
  doc: clarify
  Signed-off-by: thiswillbeyourgithub
  [email protected]
  Signed-off-by: thiswillbeyourgithub
  [email protected]
  Signed-off-by: thiswillbeyourgithub
  [email protected]
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [cb4cb51] by @thiswillbeyourgithub, 7 days ago:
  doc: clarify
  Signed-off-by: thiswillbeyourgithub
  [email protected]
  Signed-off-by: thiswillbeyourgithub
  [email protected]
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [f677e5a] by @thiswillbeyourgithub, 7 days ago:
  doc: clarify
  Signed-off-by: thiswillbeyourgithub
  [email protected]
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [39e0da5] by @thiswillbeyourgithub, 7 days ago:
  doc: clarify
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [64ca466] by @thiswillbeyourgithub (aider), 10 days ago:
  feat: Add current date to summary metadata and header to reduce hallucinations

wdoc/wdoc.py

• [4dbcf15] by @thiswillbeyourgithub, 10 days ago:
  enh: better ignoring of first line of summary if just breathing
  Signed-off-by: thiswillbeyourgithub
  [email protected]

wdoc/utils/tasks/summarize.py

Release 3.2.1

What's new

This small patch release primarily focuses on integrating OpenRouter for model pricing/metadata and refining cost calculations.

✨ Features

• Set default models to use OpenRouter ([915699c]).
• Fetch model prices and metadata automatically from OpenRouter, improving reliability ([7f840b7]).

🐛 Fixes & Enhancements

• Much improved price calculation and handling:
  • Reworked computation to account for internal thinking costs ([c0b90d8]).
  • Enhanced method for retrieving model prices, supported parameters and maximum context size ([a17b41c]).
  • Clearer error message when model prices are missing ([2b29a9d]).
• Updated litellm dependency ([179b589]).

🧪 Tests

• API integration tests now fail faster if an underlying API call fails ([9a0c856]).

Commits details since the last release

• [03aeab2] by @thiswillbeyourgithub, 2 minutes ago:
  bump version 3.2.0 -> 3.2.1

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

• [915699c] by @thiswillbeyourgithub, 6 minutes ago:
  new: set the default models to use openrouter
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md
wdoc/utils/env.py

• [c0b90d8] by @thiswillbeyourgithub, 64 minutes ago:
  fix: reworked how pricing are computed to take internal thinking into account
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/llm.py
wdoc/utils/misc.py
wdoc/utils/tasks/summarize.py
wdoc/wdoc.py

• [a17b41c] by @thiswillbeyourgithub, 80 minutes ago:
  enh: better way to get the model prices
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py
wdoc/wdoc.py

• [9a0c856] by @thiswillbeyourgithub, 22 minutes ago:
  test: crash early if one api crash fails
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/run_all_tests.sh

• [7f840b7] by @thiswillbeyourgithub, 2 hours ago:
  feat: automatically fetch the price and metadata from openrouter instead of waiting for litellm
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py
wdoc/wdoc.py

• [2b29a9d] by @thiswillbeyourgithub, 2 hours ago:
  fix: error message on missing model price
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [179b589] by @thiswillbeyourgithub, 2 hours ago:
  bump litellm version
  Signed-off-by: thiswillbeyourgithub [email protected]

setup.py

Release 3.2.0

What's new

This release focuses on improving the command-line interface (especially handling piped input/output), enhancing language model interactions (switching defaults, better caching, Ollama support), and increasing overall stability through numerous bug fixes and testing improvements.

✨ Features

• Added arguments to set specific keyword arguments (kwargs) for language models (--model_kwargs, --query_eval_model_kwargs) ([1392553]).
• Introduced WDOC_LLM_REQUEST_TIMEOUT environment variable for LLM request timeouts (default 600s), useful for Ollama ([ec3c0c5]).
• Switched default models from Claude Sonnet/Haiku to Gemini 1.5 Pro/Flash ([82ef10d]).
• Unified LLM handling to primarily use ChatLiteLLM, removing direct ChatOpenAI usage ([30a0f0c]).
• Enabled cost tracking for queries, storing the cost in the output ([e7753af]).
• Added automatic download of nltk punkt tokenizer during post-installation ([44f5bf8]).
• Overhauled Command Line Interface (CLI) argument parsing for wdoc and wdoc parse using fire ([7c51ed2], [2f4748d]).
• Removed the --pipe argument, relying on automatic stdin/stdout detection ([b03e79a], [2e6c1dd], [838f164]).
• Removed the separate wdoc_parse_file entry point; use wdoc parse instead ([2e878d2]).
• Added a new script media_url_finder.py ([beaf8fa]).

🐛 Fixes

• LLM PLACEHOLDER Caching:
  • Resolved issues with LLM caching, including invalidation when kwargs change and LangChain's SQLite cache ([cb785da], [3e3e753]).
  • Fixed edge cases in thinking block parsing for models like Gemini and updated tags (<thinking> -> <think>) ([e111bdb], [d0ae21a], [ca9245b], [99ed332]).
  • Corrected underflow errors in cost calculation due to low LLM prices ([3f18f5d], [95a1984]).
  • Addressed issues specific to Ollama: API key requirement relaxation, price assumption (zero), litellm naming (ollama_chat -> ollama), and context window estimation ([d2f92a3], [5784b25], [43c6340], [c3c15e1]).
  • Fixed handling of testing/testing models and associated parameters ([b995197], [91b5e67], [7cf840c], [9a7b95b]).
  • Fixed query_retrievers parsing ([02d7412]).
  • Pinned litellm version for stability ([1b17c78]).
• CLI PLACEHOLDER Piping:
  • Improved detection and handling of piped input/output ([2e6c1dd], [509626a], [db2fa0f]).
  • Fixed crashes and hangs when using pipes, especially with long inputs or specific test commands ([f59f34b], [414de8d], [b95b125], [826e7aa], [b6f7fd7], [177be6b]).
  • Corrected argument parsing issues affecting the --help command ([c909337]).
  • Ensured logs are not colorized and Markdown rendering is disabled when outputting to a pipe ([f1d63cd], [fe2665c]).
  • Fixed issues where debug prints or warnings were incorrectly suppressed or handled ([64fcd60], [a7724ff]).
• General:
  • Fixed various bugs in task execution, parameter handling, and attribute declarations ([27a8d35], [91d8df3], [a0eaf51], [a6effc0], [5dce2f3], [4623fcc], [b17f567], [8cc9190], [e91ed3b], [c3649ab]).
  • Corrected import path in __main__ ([0ef5e4d]).
  • Suppressed excessive INFO logs from faiss ([a17a8d1]).
  • Handled BrokenPipeError gracefully ([b40832b]).

🧪 Testing

• Improved test setup for caching, using separate directories and disabling cache where necessary ([9104f86], [89f4859], [085a87e], [6935fe7]).
• Added tests for OpenRouter/default models, piping functionality, summary/query tasks with testing models, and environment variable handling ([06e35b0], [bbb8371], [caae34c], [cb9d237], [eaafafd], [1f835eb]).
• Refactored pipe tests to use subprocess explicitly and fixed related issues (stderr redirection, pytest capture, shell usage) ([38a3571], [7f3249a], [573acf9]). (Note: Some pipe tests were later commented out ([45cf419])).

⚡ Enhancements

• Reworked logic for detecting and modifying model parameters based on the task ([564c4f9]).
• Improved load_media function to handle online media more robustly by finding and clicking appropriate buttons ([049c9cb], [67772f8], [c5828d3]).
• Added checks to prevent exceeding total token limits during summarization ([9bdcabc]).
• Refined logging levels and Markdown printing logic ([edfec82], [4ca394c], [895a60f]).

📚 Documentation

• Updated examples for Ollama arguments, model usage (Gemma -> Qwen2), and general clarity ([0087117], [49437ec], [4083dda], [404bbe4]).
• Clarified behavior related to LLM caching and model kwargs in help documentation ([c3e0219], [3e3e753], [1392553], [7db844f]).
• Updated README and help files reflecting changes in default models, CLI arguments, and entry points ([82ef10d], [b03e79a], [2e878d2], [a30bccf]).

⚙️ Build PLACEHOLDER Chore

• Bumped version to 3.2.0 ([7d69d79]).
• Added nltk to dependencies ([44f5bf8]).
• Updated .gitignore ([84aa559], [5374ee1], [39e4106], [a25e3d4]).
• Renamed embed_kwargs to embed_model_kwargs ([431efcb]).

Commits details since the last release

• [7d69d79] by @thiswillbeyourgithub, 77 seconds ago:
  bump version 3.1.0 -> 3.2.0

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

• [30a0f0c] by @thiswillbeyourgithub, 24 minutes ago:
  new: stop using both ChatOpenAI and ChatLiteLLM
  ChatLiteLLM seems to now work reliably

Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py
wdoc/docs/help.md
wdoc/utils/llm.py
wdoc/utils/retrievers.py
wdoc/utils/tasks/query.py

• [9104f86] by @thiswillbeyourgithub, 41 minutes ago:
  fix: in the pytest we should delete the cache dir regularly
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/conftest.py

• [e111bdb] by @thiswillbeyourgithub, 46 minutes ago:
  fix: fix edge case for gemini models that only end their thinking block
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [84aa559] by @thiswillbeyourgithub, 78 minutes ago:
  test: ignore cache dir
  Signed-off-by: thiswillbeyourgithub [email protected]

.gitignore

• [89f4859] by @thiswillbeyourgithub, 79 minutes ago:
  test: use a separate user dir for the cache when running the tests
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py
wdoc/utils/misc.py

• [3f18f5d] by @thiswillbeyourgithub, 79 minutes ago:
  fix: underflow error in cost
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/tasks/summarize.py
wdoc/wdoc.py

• [27a8d35] by @thiswillbeyourgithub, 2 hours ago:
  fix: latest cost attribute was not declared
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [c3c15e1] by @thiswillbeyourgithub, 2 hours ago:
  enh: if ollama is used, lower the estimate of the context window
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/llm.py

• [91d8df3] by @thiswillbeyourgithub, 2 hours ago:
  fix: wrong indentation in an if
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [a0eaf51] by @thiswillbeyourgithub, 2 hours ago:
  fix: wrong deepcopy for eval llm
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [29c9c4e] by @thiswillbeyourgithub, 2 hours ago:
  fix: test
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [cb785da] by @thiswillbeyourgithub, 2 hours ago:
  fix: make the sqlite cache already patched for langchain s stupid cache
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/customs/fix_llm_caching.py

• [3e3e753] by @thiswillbeyourgithub, 2 hours ago:
  fix: try to make it so that changing the kwargs does not reuse the cache
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md
wdoc/utils/llm.py

• [c3e0219] by @thiswillbeyourgithub, 2 hours ago:
  doc: explain that changing the kwargs will not invalidate the cache
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md

• [827d563] by @thiswillbeyourgithub, 2 hours ago:
  test: improved test to also test caching
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [e7753af] by @thiswillbeyourgithub, 2 hours ago:
  new: store the cost of the query in the output now
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [d0ae21a] by @thiswillbeyourgithub, 2 hours ago:
  fix: reworked and improved how thinking_answer_parser works
  some weak models could fail despite usable results

Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [ca9245b] by @thiswillbeyourgithub, 3 hours ago:
  fix: dont make the thinking block parser greedy
  I'm sure some models can nest thoughts

Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [99ed332] by @thiswillbeyourgithub, 3 hours ago:
  fix: most models nowadays use not
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py
wdoc/utils/prompts.py
wdoc/utils/tasks/query.py

• [085a87e] by @thiswillbeyourgithub, 3 hours ago:
  test: disable the embedding cache
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py
...

Release 3.1.0

What's new

This release primarily focuses on enhancing logging capabilities and fixing issues related to piping behavior.

Version bump to 3.1.0 ([e93dcad6]).

✨ New Features

• Logging:
  • Always display the default log location ([2fe2c431]).
  • Set log level to debug for log files and critical when used in a pipe ([130058a1]).

🚀 Enhancements

• Logging:
  • Improved log format ([61465aff], [dc06ccfd]).
  • Increased probability of early logger initialization ([01f01ac7]).
  • Clearer error messages from python-magic ([c846dafa]).

🐛 Fixes

• Piping:
  • Resolved confusion between input and output during piping ([e175b7d5]).
  • Corrected initialization of is_piped variable ([e4532d30]).
• Logging & Environment:
  • Fixed default handler issue in logger ([43c859dd]).
  • Prevented potential crash related to environment variable handling ([d3b1e2bc]).

🧹 Minor Changes

• Removed unused imports ([f3c05962]).
• Adjusted test imports structure ([69738119]).
• Removed commented code ([86b51102]).
• Removed unused disable_md_printing argument ([b3af430e]).

✅ Testing

• Added test for exception handling ([dfbfad54]).
• Added environment variable tests ([0fba8a13]).

---

Commits details since the last release

• [e93dcad] by @thiswillbeyourgithub, 10 minutes ago:
  bump version 3.0.2 -> 3.1.0

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

• [e175b7d] by @thiswillbeyourgithub, 31 minutes ago:
  fix: piping behavior was confusing input and output
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/batch_file_loader.py
wdoc/utils/env.py
wdoc/utils/loaders.py
wdoc/utils/logger.py
wdoc/utils/misc.py
wdoc/wdoc.py

• [b3af430] by @thiswillbeyourgithub, 34 minutes ago:
  forgot to remove the arg disable_md_printing
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md
wdoc/wdoc.py

• [61465af] by @thiswillbeyourgithub, 36 minutes ago:
  enh: better log format
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py

• [2fe2c43] by @thiswillbeyourgithub, 37 minutes ago:
  new: print the default log location always
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py

• [43c859d] by @thiswillbeyourgithub, 37 minutes ago:
  fix: default handler
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py

• [e4532d3] by @thiswillbeyourgithub, 47 minutes ago:
  fix: is_piped variable was wrong
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/env.py
wdoc/utils/misc.py

• [01f01ac] by @thiswillbeyourgithub, 66 minutes ago:
  enh: increase chances of logger beint initialized asap
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/init.py
wdoc/main.py

• [dc06ccf] by @thiswillbeyourgithub, 89 minutes ago:
  better log format
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py

• [f3c0596] by @thiswillbeyourgithub, 2 hours ago:
  remove unused imports
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py

• [e637c2f] by @thiswillbeyourgithub, 2 hours ago:
  new: the log level now is always at debug level for the logfile and using --debug only modifyed the stdout of user
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py

• [86b5110] by @thiswillbeyourgithub, 2 hours ago:
  minor: remove commented line
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py

• [130058a] by @thiswillbeyourgithub, 2 hours ago:
  new: if wdoc is used in a pipe, we set the log level to critical
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/env.py
wdoc/utils/logger.py

• [dfbfad5] by @thiswillbeyourgithub, 2 hours ago:
  test: add test for exception handling
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [6973811] by @thiswillbeyourgithub, 2 hours ago:
  minor: move the test imports higher up
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [0fba8a1] by @thiswillbeyourgithub, 2 hours ago:
  test: add an unexpected env variable to test that it works
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [d3b1e2b] by @thiswillbeyourgithub, 2 hours ago:
  fix: env variable handling could cause a crash
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/env.py

• [c846daf] by @thiswillbeyourgithub, 3 hours ago:
  better error message from python-magic
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/loaders.py

Release 3.0.2

What's new

Fixes

• Error Message Stability
  • Fixed a crash caused by error messages in batch file loader
    • Files affected:
      • wdoc/utils/batch_file_loader.py
      • wdoc/utils/loaders.py
  • Commit hash: [4af7dc6]
  • Author: @thiswillbeyourgithub

Version Bump

• Version Update
  • Updated version from 3.0.1 to 3.0.2 for better stability and minor enhancements
    • Files affected for version bump:
      • bumpver.toml
      • docs/source/conf.py
      • setup.py
      • wdoc/wdoc.py
  • Commit hash: [504b5c9]
  • Author: @thiswillbeyourgithub

Note

These updates aim to enhance overall functionality and prevent errors from causing interrupts, ensuring a smoother user experience. The version bump signifies an incremental improvement with significant internal fixes.

Commits details since the last release

• [504b5c9] by @thiswillbeyourgithub, 6 seconds ago:
  bump version 3.0.1 -> 3.0.2

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

• [4af7dc6] by @thiswillbeyourgithub, 10 seconds ago:
  fix: error message was causing a crash
  Signed-off-by: thiswillbeyourgithub
  [email protected]
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/batch_file_loader.py
wdoc/utils/loaders.py

Release 3.0.1

What's new

Version 3.0.1 - April 18, 2025

Chores Housekeeping

• Version Bump:
  • Bumped version from 3.0.0 to 3.0.1.
    • Commit: [3341823] by @thiswillbeyourgithub

Bug Fixes

• Error Message Fix:
  • Resolved issue where error message was causing a crash.
    • Location: wdoc/utils/loaders.py
    • Commit: [20b5ccd] by @thiswillbeyourgithub

Documentation

• Companion Tool Mention:
  • Updated README to mention that a companion tool might be needed.
    • Location: README.md
    • Commit: [75bc42c] by @thiswillbeyourgithub

Testing

• Test Script Modification:
  • Changed script to use rm instead of trash.
    • Location: tests/run_all_tests.sh
    • Commit: [75a21ee] by @thiswillbeyourgithub

---

Commits details since the last release

• [3341823] by @thiswillbeyourgithub, 6 seconds ago:
  bump version 3.0.0 -> 3.0.1

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

• [20b5ccd] by @thiswillbeyourgithub, 44 seconds ago:
  fix: error message was causing a crash
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/loaders.py

• [75bc42c] by @thiswillbeyourgithub, 16 hours ago:
  doc: mention the companion tool might be needed
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [75a21ee] by @thiswillbeyourgithub, 18 hours ago:
  test: use rm instead of trash
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/run_all_tests.sh

Release 3.0.0

What's new

• Version Update 3.0.0
• Environment Variable Handling
  • Completely revamped the approach to handling environment variables.
    • Commits: [fe49bd9], [02d2f84]
• Logging Improvements
  • Integrated loguru for more comprehensive logging.
    • Commits: [31ade20]
  • Altered default log level based on debug status.
    • Commits: [2e9b7f8]

Enhancements

• Documentation and Scripts
  • Updated README and other documentation to reflect recent changes.
    • Commits: [247ef4d], [11cc311], [e9fa7c1]
  • Improved test scripts for more efficient testing.
    • Commits: [94963ef]
• Code Optimization
  • Streamlined environment variable checking for redundancy optimization.
    • Commits: [3f6fcd6], [d4fd1d7]
  • Various small optimizations across multiple files led to better semantic batch handling.
    • Commits: [04d2b92], [6ac351f], [cb1024d]

Commits details since the last release

• [51bbc55] by @thiswillbeyourgithub, 29 minutes ago:
  bump version 2.9.0 -> 3.0.0

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

• [94963ef] by @thiswillbeyourgithub, 21 minutes ago:
  better test script
  Signed-off-by: thiswillbeyourgithub
  [email protected]

tests/run_all_tests.sh

• [247ef4d] by @thiswillbeyourgithub, 30 minutes ago:
  doc: update todo list
  Signed-off-by: thiswillbeyourgithub
  [email protected]

README.md

• [8f516e4] by @thiswillbeyourgithub, 45 minutes ago:
  fix: wrongly setting env vars to True instead of "true"
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/env.py
wdoc/wdoc.py

• [3491b6a] by @thiswillbeyourgithub, 47 minutes ago:
  fix: main was still using flags instead of env
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/main.py

• [c06ec64] by @thiswillbeyourgithub, 62 minutes ago:
  new: compulsively check for unexpected values in env var
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/env.py

• [f216546] by @thiswillbeyourgithub, 67 minutes ago:
  replace a print by a logger.warning
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/env.py

• [f133503] by @thiswillbeyourgithub, 70 minutes ago:
  remove weird handling of md_printing_disabled
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py

• [138ef37] by @thiswillbeyourgithub, 71 minutes ago:
  use loguru in main instead of print
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/main.py

• [d4fd1d7] by @thiswillbeyourgithub, 73 minutes ago:
  new: stop using flags.py to store something that should be stored in env.py
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md
wdoc/utils/init.py
wdoc/utils/batch_file_loader.py
wdoc/utils/embeddings.py
wdoc/utils/env.py
wdoc/utils/flags.py
wdoc/utils/llm.py
wdoc/utils/loaders.py
wdoc/utils/misc.py
wdoc/utils/tasks/query.py
wdoc/wdoc.py

• [a06b09f] by @thiswillbeyourgithub, 2 hours ago:
  minor: explanatory comment
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/env.py

• [b9904af] by @thiswillbeyourgithub, 2 hours ago:
  typo
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md

• [177e81a] by @thiswillbeyourgithub, 2 hours ago:
  fix: unbounlocalerror incomprehenssible unless I reimport logger
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [69a6013] by @thiswillbeyourgithub, 2 hours ago:
  minor: move cache dir declaration misc.py
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py
wdoc/utils/misc.py
wdoc/wdoc.py

• [2e9b7f8] by @thiswillbeyourgithub, 2 hours ago:
  swtich default log level depending on if is_debug is set
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py

• [31ade20] by @thiswillbeyourgithub, 2 hours ago:
  feat: switch logging backend to loguru
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/main.py
wdoc/utils/batch_file_loader.py
wdoc/utils/embeddings.py
wdoc/utils/interact.py
wdoc/utils/llm.py
wdoc/utils/loaders.py
wdoc/utils/logger.py
wdoc/utils/misc.py
wdoc/utils/prompts.py
wdoc/utils/tasks/query.py
wdoc/utils/tasks/summarize.py
wdoc/wdoc.py

• [d362034] by @thiswillbeyourgithub, 3 hours ago:
  minor: pass the youtube playlist title metadata to docs
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/batch_file_loader.py

• [bd100cc] by @thiswillbeyourgithub, 3 hours ago:
  test: add an api test that set debug to True
  Signed-off-by: thiswillbeyourgithub
  [email protected]

tests/test_wdoc.py

• [6ac351f] by @thiswillbeyourgithub, 3 hours ago:
  fix: remove loaders_temp_dir_file contraption that was causing issue in some parallel edge cases
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/batch_file_loader.py
wdoc/utils/loaders.py
wdoc/utils/misc.py

• [c7e440d] by @thiswillbeyourgithub, 4 hours ago:
  fix: error in edge case in summary
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/tasks/summarize.py

• [7087501] by @thiswillbeyourgithub, 4 hours ago:
  test: better test by checking if the default install works too
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/run_all_tests.sh

• [2dda653] by @thiswillbeyourgithub, 4 hours ago:
  update tests to use the new env handling
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [04d2b92] by @thiswillbeyourgithub, 4 hours ago:
  remove experimental var test
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [e9fa7c1] by @thiswillbeyourgithub (aider), 4 hours ago:
  docs: add docstrings to env.py

wdoc/utils/env.py

• [11cc311] by @thiswillbeyourgithub, 4 hours ago:
  doc: add explanation for runtime modification of env variables
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [fdc237f] by @thiswillbeyourgithub, 4 hours ago:
  update scripts for the env variable handling
  Signed-off-by: thiswillbeyourgithub [email protected]

scripts/AnkiFiltered/AnkiFilteredDeckCreator.py

• [4ce5d09] by @thiswillbeyourgithub, 4 hours ago:
  finish updating how env variables are handled including for managing the faiss scoring function
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/embeddings.py

• [d28e7e0] by @thiswillbeyourgithub, 4 hours ago:
  new: replace LANGFUSE env var by wdoc's LANGFUSE var twice instead of just at runtime, just in case
  Signed-off-by: thiswillbeyourgithub
  [email protected]

wdoc/utils/env.py
wdoc/utils/misc.py

• [02d2f84] by @thiswillbeyourgithub, 5 hours ago:
  new: crash if WDOC_LANGFUSE keys are set in env but langfuse fails to import
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [b037236] by @thiswillbeyourgithub, 5 hours ago:
  more updating of the env handling
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/batch_file_loader.py
wdoc/utils/import_tricks.py
wdoc/utils/llm.py
wdoc/utils/loaders.py
wdoc/utils/misc.py
wdoc/utils/tasks/query.py
wdoc/utils/typechecker.py
wdoc/wdoc.py

• [0266909] by @thiswillbeyourgithub, 5 hours ago:
  minor: remove unused import
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/interact.py
wdoc/utils/logger.py

• [3f6fcd6] by @thiswillbeyourgithub, 5 hours ago:
  the private env var checker is now redundant so removed
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/flags.py

• [d15fcd3] by @thiswillbeyourgithub, 5 hours ago:
  fix: better way to check if asyncio nested patch should be applied or not
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/init.py

• [fe49bd9] by @thiswillbeyourgithub, 5 hours ago:
  new: completely changed how env variables are handled
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/env.py
wdoc/utils/errors.py

• [bb52773] by @thiswillbeyourgithub, 13 days ago:
  new: rename 'default' argument of query_retrievers to 'basic' because its not the default
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md
wdoc/docs/help.md
wdoc/wdoc.py

• [3c47a57] by @thiswillbeyourgithub, 13 days ago:
  docs: remove obsolete line about hyde retriever
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md

• [20d8499] by @thiswillbeyourgithub, 2 weeks ago:
  update todo
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [4c4b0d8] by @thiswillbeyourgithub, 2 weeks ago:
  update todo
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [cb1024d] by @thiswillbeyourgithub, 2 weeks ago:
  new: increase the max ac...

Release 2.9.0

What's new

New Features

• Shell Command Testing
  • Added shell command test for NYTimes parsing with content validation. [d3832f2]

Fixes

• Intermediate Merging
  • Fixed error during batch merging, ensuring intermediate answers are handled properly when too large. [d06cbb3]
  • Improved logic for retry attempts during batch merging. [573e15f]
• Parsing Reliability
  • Prevent return of intermediately parsed output on parsing failure. [78c9364]
  • Adapted handling to retry when encountering unparseable output or finish_reason=length. [763e9b4]
• Backend and Output
  • Resolved backend errors and edge case issues breaking summary generation. [bf00ced, 89c01de]
  • Corrected handling of test outputs to avoid crashing in specific import modes. [ee416ec]
• Testing Corrections
  • Corrected various test functions and ensured expected outcomes align with API changes. [bfcba71, 89c01de, 292ce90]

Documentation

• General Updates
  • Enhanced walkthrough formatting for improved readability in documentation. [2ec7fad]
  • Provided context about the origins of wdoc in README. [31e6c5d]
• Example and Help Docs
  • Updated examples documentation to remove confusing import_mode arguments. [cef5cdf]
  • Improved query and summary help documentation to reflect recent changes. [a289759, f1a6294]

Improvements

• Configuration and Setup Adjustments
  • Simplified the post-install script to handle dependencies via uv if present. [7a55143, 4ed0a35]
• Performance and Debugging Enhancements
  • Bumped max_tokens for intermediate answers to accommodate larger outputs during processing. [df059b6]
  • Stored original strings before parsing for effective debugging. [6ec3957]
  • Addressed deprecation warnings to keep up with latest standards. [15e4793]

Minor Changes

• Code and Debug Tune-ups
  • Streamlined testing arguments and outputs for precise validation. [7e5e4ce, b7f1a1f, d79b4c5]
  • Fine-tuned post-install scripts and functional debug outputs for better clarity. [d620f87]
• Enhanced wdoc Docs Via SVG Files (WIP)
  • Create SVG diagram and documentation for summary algorithm. [b5f49a9]
  • Added SVG visualization and improved design and intuitive flow representation for better understanding. [9489a8a, 703dde0]

Commits details since the last release

• [d06cbb3] by @thiswillbeyourgithub, 34 minutes ago:
  fix: error when merging batch when intermediate answers got so large the model can't merge them anymore
  We just concatennate them using semantic order and that will be good
  enough, the alternative is too expensive

Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [573e15f] by @thiswillbeyourgithub, 35 minutes ago:
  fix: one more trial given to merge batches
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [7edbe1f] by @thiswillbeyourgithub, 54 minutes ago:
  doc: add helpful debug message if abrupt message tail
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [df059b6] by @thiswillbeyourgithub, 55 minutes ago:
  new: bump max_token for intermediate answer from 1000 to 4000
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [78c9364] by @thiswillbeyourgithub, 3 hours ago:
  fix: don't return intermediately parsed output if parsing fails
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [6ec3957] by @thiswillbeyourgithub, 3 hours ago:
  minor: store the original string before parsing to help debugging
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [ee1c857] by @thiswillbeyourgithub, 3 hours ago:
  minor: better order of the output price prints
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [cc49037] by @thiswillbeyourgithub, 3 hours ago:
  fix: out_file test
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [cef5cdf] by @thiswillbeyourgithub, 3 hours ago:
  fix: forgot to remove import_mode args from examples
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/examples.md

• [d3832f2] by @thiswillbeyourgithub (aider), 3 hours ago:
  feat: Add shell command test for NYTimes parsing with content validation

tests/test_wdoc.py

• [ee416ec] by @thiswillbeyourgithub, 3 hours ago:
  new: don't crash if using import_mode at the same time as out_file
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [32ecdbd] by @thiswillbeyourgithub, 4 hours ago:
  test: remove unused debug and verbose argsc
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [a13d20a] by @thiswillbeyourgithub, 4 hours ago:
  new: remove confusing arg 'import_mode' and set it automatically depending on if imported or launched from cli
  Signed-off-by: thiswillbeyourgithub
  [email protected]

README.md
scripts/AnkiFiltered/AnkiFilteredDeckCreator.py
scripts/TheFiche/TheFiche.py
tests/test_wdoc.py
wdoc/main.py
wdoc/docs/help.md
wdoc/wdoc.py

• [763e9b4] by @thiswillbeyourgithub, 4 hours ago:
  fix: now if eval_llm returns something unparsable or with finish_reason=length we always retry
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [15e4793] by @thiswillbeyourgithub, 4 hours ago:
  minor: address deprecation warnings
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/tasks/query.py

• [b7f1a1f] by @thiswillbeyourgithub, 4 hours ago:
  test: set semantic batching test to api mark
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [7e5e4ce] by @thiswillbeyourgithub, 4 hours ago:
  test: move semantic batching test to the api section
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [e5530b6] by @thiswillbeyourgithub, 5 hours ago:
  test: add test for mistral embeddings
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [d79b4c5] by @thiswillbeyourgithub, 5 hours ago:
  fix: remove unused arg in tests
  Signed-off-by: thiswillbeyourgithub
  [email protected]
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [13ad2aa] by @thiswillbeyourgithub, 5 hours ago:
  test: ollama should be an api mark not basic
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [d3679b3] by @thiswillbeyourgithub, 5 hours ago:
  minor: sort pytest by mark
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [bfcba71] by @thiswillbeyourgithub, 6 hours ago:
  fix test
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [292ce90] by @thiswillbeyourgithub, 6 hours ago:
  fix: test of query
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [740dc25] by @thiswillbeyourgithub, 6 hours ago:
  fix: test of out_file
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [ff73711] by @thiswillbeyourgithub, 6 hours ago:
  fix: summary test
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [3b292ea] by @thiswillbeyourgithub, 6 hours ago:
  fix: remove unused arg in tests
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/test_wdoc.py

• [bf00ced] by @thiswillbeyourgithub, 6 hours ago:
  fix: edge case was breaking summary
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [89c01de] by @thiswillbeyourgithub, 6 hours ago:
  fix: backend error in one edge case
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [ebcf92f] by @thiswillbeyourgithub (aider), 6 hours ago:
  feat: Change default query relevancy threshold to -0.5

wdoc/docs/help.md
wdoc/wdoc.py

• [443aab4] by @thiswillbeyourgithub, 7 hours ago:
  fix: query_task arg is actually optional
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [5847cd3] by @thiswillbeyourgithub, 7 hours ago:
  fix: missing var if only one document present
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [31e6c5d] by @thiswillbeyourgithub (aider), 12 hours ago:
  docs: Add context about medical student's motivation for creating wdoc

README.md

• [2ec7fad] by @thiswillbeyourgithub (aider), 14 hours ago:
  style: Update walkthrough formatting to use triple backticks for code blocks

wdoc/docs/examples.md

• [c06a849] by @thiswillbeyourgithub, 15 hours ago:
  update roadmap
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [7d1ba8b] by @thiswillbeyourgithub, 15 hours ago:
  fix: link to examples
  Signed-off-by: thiswillbeyourgithub [email protected]

README.md

• [d003d8e] by @thiswillbeyourgithub, 25 hours ago:
  fix: ongoing fix for the summary test
  Signed-off-b...

Release 2.8.0

What's new

New Features

• Add an environment variable to control invalid document evaluation behavior
• Add WDOC_APPLY_ASYNCIO_PATCH env variable to manage asyncio patching
• Specify name of LocalFileStore for better logging
• Add a decorator for more useful debug logs

Improvements

Logging

• Introduced better format for debug prints
• Harmonized default environment value presentation in the documentation

Refactoring

• Replaced hash-based source identifiers with a consistent format
• Improved source identifier handling for single document cases

Bug Fixes

• MAJOR Resolved error where sources were not properly referenced
• MAJOR Addressed problems with cluster detection in text analysis
• Applied patch before running tests to resolve buggy processes
• Corrected issues with concurrency setting causing unexpected behavior
• Fixed an obsolete script referencing an outdated environment variable

Documentation

• Removed outdated mention of winston doc, replacing with current references

Dependency Management

• To run tests, pytest-xdist must be installed
• Bumped PersistDict to the latest version

Minor Changes

• Various minor code and logic corrections throughout the codebase

Commits details since the last release

• [bf143b9] by @thiswillbeyourgithub, 16 minutes ago:
  bump version 2.7.1 -> 2.8.0

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

• [f97935e] by @thiswillbeyourgithub, 3 minutes ago:
  Revert "tests: remove the fixture from tests as they are bugging some tests"
  This reverts commit 34adb42.

tests/conftest.py

• [6515373] by @thiswillbeyourgithub, 6 minutes ago:
  fix: apply patch before running tests
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/conftest.py
tests/run_all_tests.sh

• [34adb42] by @thiswillbeyourgithub, 14 minutes ago:
  tests: remove the fixture from tests as they are bugging some tests
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/conftest.py

• [10633a6] by @thiswillbeyourgithub, 61 minutes ago:
  fix: cant use xdist for the api tests apparently
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/run_all_tests.sh

• [5e7233b] by @thiswillbeyourgithub, 77 minutes ago:
  fix: obsolete script was using an old import env var name
  Signed-off-by: thiswillbeyourgithub [email protected]

scripts/AnkiFiltered/AnkiFilteredDeckCreator.py

• [7265e00] by @thiswillbeyourgithub, 2 hours ago:
  fix: to run the tests we must install pytest-xdist
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/run_all_tests.sh

• [ed11934] by @thiswillbeyourgithub, 2 hours ago:
  fix: to run the tests we need to patch asyncio
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/run_all_tests.sh

• [f382e17] by @thiswillbeyourgithub, 4 hours ago:
  new: better format for debug prints
  Signed-off-by: thiswillbeyourgithub
  [email protected]

wdoc/utils/logger.py

• [e213351] by @thiswillbeyourgithub, 4 hours ago:
  minor: remove a mention of winston doc and replace by wdoc
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md

• [11b5c95] by @thiswillbeyourgithub, 4 hours ago:
  fix: set default concurrency to 1 actually because it is causing issues
  Signed-off-by: thiswillbeyourgithub
  [email protected]

wdoc/docs/help.md
wdoc/utils/env.py

• [dca2b47] by @thiswillbeyourgithub (aider), 7 hours ago:
  feat: Add environment variable to control invalid document evaluation behavior

wdoc/docs/help.md
wdoc/utils/env.py
wdoc/utils/tasks/query.py

• [ffa2d67] by @thiswillbeyourgithub, 7 hours ago:
  docs: harmonize default env valuee presentation
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md

• [95ec3aa] by @thiswillbeyourgithub, 7 hours ago:
  fix: set default llm concurrency to 5 instead of 15
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md
wdoc/utils/env.py

• [c056ba4] by @thiswillbeyourgithub, 7 hours ago:
  fix: exit code should have been 0 not 1
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [26ba02b] by @thiswillbeyourgithub, 7 hours ago:
  fix: litellm debugging
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [9ac4f32] by @thiswillbeyourgithub, 7 hours ago:
  typo
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/tasks/query.py

• [3d3006e] by @thiswillbeyourgithub (aider), 31 hours ago:
  feat: Add WDOC_APPLY_ASYNCIO_PATCH env variable to control asyncio patching

wdoc/init.py
wdoc/docs/help.md
wdoc/utils/env.py

• [dfccef0] by @thiswillbeyourgithub, 31 hours ago:
  minor
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/init.py

• [ed5590b] by @thiswillbeyourgithub, 32 hours ago:
  new: reworked how we parse the outputs of eval_llm
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/tasks/query.py

• [f8cd452] by @thiswillbeyourgithub, 32 hours ago:
  new: use more debug logging
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/embeddings.py
wdoc/utils/llm.py
wdoc/utils/tasks/query.py
wdoc/wdoc.py

• [796f8cf] by @thiswillbeyourgithub, 32 hours ago:
  major fix: finally fix the bug where some sources were not properly referenced
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py
wdoc/utils/prompts.py
wdoc/utils/tasks/query.py
wdoc/wdoc.py

• [6538910] by @thiswillbeyourgithub, 33 hours ago:
  new: actually no need to use a debug print for the md printing
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py

• [3201936] by @thiswillbeyourgithub, 34 hours ago:
  fix: do not use info level debug print for the final answer of query
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py

• [357925f] by @thiswillbeyourgithub, 34 hours ago:
  fix: in rare cases where the clustering only finds 1 cluster

wdoc/utils/tasks/query.py

• [072c00d] by @thiswillbeyourgithub, 34 hours ago:
  new: allow finding oonly 2 clusters in a group of texts
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/tasks/query.py

• [1786ff7] by @thiswillbeyourgithub, 34 hours ago:
  fix: avoid issues when asking for more clusters than texts
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/tasks/query.py

• [8238599] by @thiswillbeyourgithub, 35 hours ago:
  bump persistdict
  Signed-off-by: thiswillbeyourgithub [email protected]

setup.py

• [fd01302] by @thiswillbeyourgithub, 2 days ago:
  new: add a decorator to get more useful debug logs
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py
wdoc/utils/tasks/query.py
wdoc/utils/tasks/summarize.py

• [4b58b16] by @thiswillbeyourgithub, 2 days ago:
  add more debug logs
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [971cda5] by @thiswillbeyourgithub, 2 days ago:
  apply black
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [639e443] by @thiswillbeyourgithub, 2 days ago:
  fix: use the up to date way to increase litellm log level

wdoc/wdoc.py

• [ef7cb6b] by @thiswillbeyourgithub, 2 days ago:
  new: add more debug prints
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/embeddings.py

• [dc5d203] by @thiswillbeyourgithub, 2 days ago:
  remove unused status function
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/embeddings.py

• [d836e6a] by @thiswillbeyourgithub, 2 days ago:
  new: set USER_AGENT before importing wdoc and its dep

wdoc/init.py
wdoc/wdoc.py

• [5b24a9f] by @thiswillbeyourgithub, 2 days ago:
  bump persistdict
  Signed-off-by: thiswillbeyourgithub [email protected]

setup.py

• [55ac32b] by @thiswillbeyourgithub, 2 days ago:
  update gitignore
  Signed-off-by: thiswillbeyourgithub
  [email protected]

.gitignore

• [4a75037] by @thiswillbeyourgithub, 2 days ago:
  new: specify name of LocalFileStore
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/customs/fix_llm_caching.py
wdoc/utils/embeddings.py
wdoc/utils/retrievers.py

• [55d0396] by @thiswillbeyourgithub, 2 days ago:
  new: use my custom LocalFileStore for retrievers too
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/retrievers.py

• [8b5269d] by @thiswillbeyourgithub, 2 days ago:
  bump PersistDict
  Signed-off-by: thiswillbeyourgithub [email protected]

setup.py

• [84850f7] by @thiswillbeyourgithub, 2 days ago:
  new: add a name to the LocalFileStore for better logging
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/customs/compressed_embeddings_cacher.py

• [c379df4] by @thiswillbeyourgithub, 2 days ago:
  fix: t...

Release 2.7.1

What's new

Enhancements

• Performance Improvements:
  • Updated PersistDict to the latest version for better performance.
  • Improved the efficiency of embeddings by attempting in-memory caching during load.
• Model Handling:
  • Introduced adjustments to avoid reliance on external sources like litellm for model parameters and maximum tokens.
  • Implemented memoization for frequently called model name handlers to reduce computation times.
• Testing Experience:
  • Integrated pytest-xdist to parallelize and speed up the test execution.
  • Optimized test performance by utilizing a session fixture scope.

Bug Fixes

• Performance Fine-tuning:
  • Corrected versioning discrepancies for incrementals.
  • Addressed issues with multiple output answer handling.
• Logging and Debugging:
  • Enhanced debug message levels to provide clearer output.
  • Resolved issues with redundant logger information.
• Model Compatibility:
  • Amended model parameter defaults where strict support was unavailable.
  • Improved the hash method in ModelName for cache support integrity.
  • Fixed exit value inconsistency in create_embeddings.

Minor Changes

• Code Refinement:
  • Eliminated unused arguments and flags for simplified code.
  • Enhanced error messages for better clarity during debugging.

Commits details since the last release

• [5679fd9] by @thiswillbeyourgithub, 7 minutes ago:
  bump PersistDict to latest version for perf
  Signed-off-by: thiswillbeyourgithub
  [email protected]

setup.py

• [63d3386] by @thiswillbeyourgithub, 60 minutes ago:
  bump version 2.7.0 -> 2.7.1

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

• [78faa58] by @thiswillbeyourgithub, 34 minutes ago:
  fix: use latest verdion of persistic for better performance
  Signed-off-by: thiswillbeyourgithub [email protected]

setup.py

• [9ff9176] by @thiswillbeyourgithub, 55 minutes ago:
  minor: the message to start langfuse shouod be in debug level
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [5f21bc9] by @thiswillbeyourgithub, 69 minutes ago:
  new: no need actually to print message when bypassinh model name matching
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [a37e732] by @thiswillbeyourgithub, 71 minutes ago:
  fix: handle multiple output answers

wdoc/wdoc.py

• [9cd6407] by @thiswillbeyourgithub, 4 hours ago:
  new: try to cache embeddings in memory when loading them
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/embeddings.py
wdoc/utils/misc.py

• [9b11e82] by @thiswillbeyourgithub, 4 hours ago:
  new: use a debug print when bypassing a modelname matching
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [5cfa86a] by @thiswillbeyourgithub, 4 hours ago:
  new: update the model parameters if it actually does not support n
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [5202b45] by @thiswillbeyourgithub, 4 hours ago:
  fix: better handling of models that do not support modifying the n parameter
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md
wdoc/wdoc.py

• [9d1e4f6] by @thiswillbeyourgithub, 4 hours ago:
  fix: add hash method to ModelName to make it work with caches
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [72f6409] by @thiswillbeyourgithub, 4 hours ago:
  new: try to avoid trusting litellm or openrouter for model parameters and max_tokens
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [d689b6a] by @thiswillbeyourgithub, 4 hours ago:
  new: add memoize decorator to frequently called model name handlers
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [21108bd] by @thiswillbeyourgithub, 4 hours ago:
  minor: forgot a type checker decorator
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/misc.py

• [ffa0f80] by @thiswillbeyourgithub, 5 hours ago:
  minor: better error message
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

• [ec8682f] by @thiswillbeyourgithub, 5 hours ago:
  fix: forgot to update an exit value in create_embeddings
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/embeddings.py

• [6933655] by @thiswillbeyourgithub, 3 days ago:
  use pytest-xdist when running all tests
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/run_all_tests.sh

• [371ed6c] by @thiswillbeyourgithub, 3 days ago:
  add pytest xdist to setup
  Signed-off-by: thiswillbeyourgithub [email protected]

setup.py

• [3bb7596] by @thiswillbeyourgithub, 3 days ago:
  test: use scope session fixture for pytest to make them faster
  Signed-off-by: thiswillbeyourgithub [email protected]

tests/conftest.py

• [00c46ca] by @thiswillbeyourgithub, 3 days ago:
  fix: forgot to remove a logger.info duplicate line
  Signed-off-by: thiswillbeyourgithub
  [email protected]

wdoc/utils/logger.py

• [49fcb23] by @thiswillbeyourgithub, 3 days ago:
  remove the is_silent argument and flag that are not used anymore
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/help.md
wdoc/utils/flags.py
wdoc/utils/logger.py

• [3882630] by @thiswillbeyourgithub, 3 days ago:
  new: use the logger instead of tqdm write to avoid duplicate prints and have better importability
  Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/logger.py