Merged
Conversation
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5.6.0 to 6.0.0. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v5.6.0...v6.0.0) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
v0.7.1 updates
Za/adversarial generator
171 adversarial generator
…elop/actions/setup-python-6.0.0 chore(deps): bump actions/setup-python from 5.6.0 to 6.0.0
add unit tests for adversarial generator
* completed changes for progress bar implementation * added try catch to stop progress for execption * added progress bar description * added progressbar parameters for all notebooks * fixed linting and formatting errors * completed progress bar - whitespace issue and gender attrib issue * fixed linting * update attribute info in notebook * reverting old notebook metadata * reverting metadata for counterfactual metrics * refactored message based on comments * added attributes for response generator * reverting changes * added attribute info and removed print statement * added attribute and removed print statement * added attribute info * completed progress bar for toxicity * completed changes for stereotype metrics progress bar * fixed bugs and linting issues * fixed output indentation issues * updated poetry lock file * updated attributions and info on example notebooks * bug fixes * updated poetry file and add rich * fixed linting issues * deprecate estimate_token_cost * fix toxicity progress bars * update progress bars for COBS * update progress bars for SA * create progress bar utilities * use progress utilities with toxicity * polish progress bars for counterfactual * polish progress bars for stereotype * polish progress bars for toxicity * polish progress bars for generator classes * update notebooks * fix logic when progress bar not displayed * add rich dependency * ruff changes * progress bar updates * polish autoeval progress bars * ruff changes * update progress bar for adversarial toxicity and sentiment bug * update uni tests * add disk space cleanup to github workflow * add unit tests for progress bar * update unit test * disable live rendering in CI * update unit test * update unit test * update ci.yaml * add conftest.py * mock progress bar to fix LiveError * mock progress bar to fix Rich Liv eError * mock progress bar * update unit tests * update unit tests * update ci.yaml * update progress bar completion * fix progress bars for adversarial --------- Co-authored-by: Jinesh Mehta <mehtajineshs@gmail.com> Co-authored-by: Zeya Ahmad <za2291@columbia.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This release adds two new features:
Progress bars with
richProgress tracking is now available for all metric computation and generation steps using
richprogress bars. For the applicable methods, these progress bars can be turned on/off using theshow_progress_barboolean parameter.Adversarial evaluations
We now offer two out-of-the-box adversarial evaluations with LangFair, for scenarios when users wish to explore worst-case generations for their LLM of choice. These evaluations, which include toxicity and counterfactual fairness assessments, are implemented using the
AdversarialGeneratorclass.1. Adversarial toxicity evaluation
This implementation generates responses with the provided LLM to prompts from the RealToxicityPrompts dataset.
2. Adversarial counterfactual evaluation
This implementation generates counterfactual responses with the provided LLM to adversarially designed counterfactual prompt sets. Off-the-shelf templates are available for a wide variety of attributes, including gender, race, age, etc.