Skip to content

Commit d367798

Browse files
dylanboucharddylanbouchard
authored
Merge pull request #350 from cvs-health/patch/v0.5.5
Patch release: `v0.5.5`
2 parents 84a6121 + eefba70 commit d367798

File tree

19 files changed

+6645
-7240
lines changed

19 files changed

+6645
-7240
lines changed

‎.github/workflows/ci.yaml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,12 @@ on:
55
paths:
66
- 'uqlm/**'
77
- 'tests/**'
8-
- 'poetry.lock'
8+
- 'uv.lock'
99
pull_request:
1010
paths:
1111
- 'uqlm/**'
1212
- 'tests/**'
13-
- 'poetry.lock'
13+
- 'uv.lock'
1414

1515
jobs:
1616
run-tests:
@@ -31,13 +31,13 @@ jobs:
3131
- name: Checkout code
3232
uses: actions/checkout@v5
3333

34-
- name: Set up Python
35-
uses: actions/setup-python@v5
34+
- name: Set up uv
35+
uses: astral-sh/setup-uv@v5
3636
with:
37-
python-version: ${{matrix.python-version}}
37+
python-version: ${{ matrix.python-version }}
3838

3939
- name: Install dependencies
40-
run: python -m pip install pytest pytest-asyncio pytest-rerunfailures langchain-openai .
40+
run: uv sync --group test
4141

4242
- name: Run tests
43-
run: pytest -v
43+
run: uv run pytest -v

‎.github/workflows/documentation.yaml‎

Lines changed: 6 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -42,14 +42,10 @@ jobs:
4242
sed -i "s/^release = .*/release = '${{ steps.version.outputs.clean_version }}'/" docs/source/conf.py
4343
head -n 20 docs/source/conf.py
4444
45-
- name: Set up Python
46-
uses: actions/setup-python@v5
45+
- name: Set up uv
46+
uses: astral-sh/setup-uv@v5
4747
with:
48-
python-version: 3.12
49-
50-
- name: Install Poetry
51-
run: |
52-
pip install poetry
48+
python-version: "3.12"
5349

5450
- name: Download and install Pandoc
5551
run: |
@@ -66,10 +62,7 @@ jobs:
6662
run: pandoc --version
6763

6864
- name: Install dependencies
69-
run: |
70-
poetry lock
71-
poetry install --with docs
72-
eval $(poetry env activate)
65+
run: uv sync --group docs
7366

7467
- name: Checkout gh-pages branch to get versions.json
7568
uses: actions/checkout@v4
@@ -87,7 +80,7 @@ jobs:
8780
8881
- name: Build Sphinx docs
8982
run: |
90-
eval $(poetry env activate)
83+
source .venv/bin/activate
9184
make -C docs clean
9285
make -C docs html
9386
@@ -112,4 +105,4 @@ jobs:
112105
github_token: ${{ secrets.GITHUB_TOKEN }}
113106
publish_dir: ./docs/build/html
114107
destination_dir: latest
115-
keep_files: true
108+
keep_files: true

‎.github/workflows/linting.yml‎

Lines changed: 7 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -16,20 +16,13 @@ jobs:
1616
runs-on: ubuntu-latest
1717
steps:
1818
- uses: actions/checkout@v4
19-
- name: Set up Python
20-
uses: actions/setup-python@v5
19+
- name: Set up uv
20+
uses: astral-sh/setup-uv@v5
2121
with:
22-
python-version: "3.9"
23-
cache: 'pip'
24-
- name: Get Ruff version and install
25-
run: |
26-
pip install poetry
27-
RUFF_VERSION=$(poetry show --only=dev | grep '^ruff ' | awk '{print $3}')
28-
echo "Installing ruff==$RUFF_VERSION"
29-
pip install ruff==$RUFF_VERSION
22+
python-version: "3.12"
23+
- name: Install dev dependencies
24+
run: uv sync --group dev --no-install-project
3025
- name: Lint with Ruff
31-
run: |
32-
ruff check uqlm/
26+
run: uv run ruff check uqlm/
3327
- name: Check for unformatted files
34-
run: |
35-
ruff format --check uqlm/
28+
run: uv run ruff format --check uqlm/

‎CONTRIBUTING.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,8 +49,8 @@ Please ensure your pull request adheres to the following guidelines:
4949
1. Clone the repository: `git clone https://github.com/cvs-health/uqlm.git`
5050
2. Navigate to the project directory: `cd uqlm`
5151
3. Create and activate a virtual environment (using `venv` or `conda`)
52-
4. Install poetry (if you don't already have it): `pip install poetry`
53-
5. Install uqlm with dev dependencies: `poetry install --with dev`
52+
4. Install uv (if you don't already have it): `pip install uv` or `curl -LsSf https://astral.sh/uv/install.sh | sh`
53+
5. Install uqlm with dev dependencies: `uv sync --group dev`
5454
6. Install our pre-commit hooks to ensure code style compliance: `pre-commit install`
5555
7. Run tests to ensure everything is working: `pre-commit run --all-files`
5656

‎README.md‎

Lines changed: 61 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -6,14 +6,44 @@
66
</picture>
77
</p>
88

9+
<h1 align="center">uqlm: Uncertainty Quantification for Language Models</h1>
910

10-
# uqlm: Uncertainty Quantification for Language Models
11-
12-
[![Build Status](https://github.com/cvs-health/uqlm/actions/workflows/ci.yaml/badge.svg)](https://github.com/cvs-health/uqlm/actions)
13-
[![PyPI version](https://badge.fury.io/py/uqlm.svg)](https://pypi.org/project/uqlm/)
14-
[![Documentation Status](https://img.shields.io/badge/docs-latest-blue.svg)](https://cvs-health.github.io/uqlm/latest/index.html)
15-
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
16-
[![](https://img.shields.io/badge/arXiv-2504.19254-B31B1B.svg)](https://arxiv.org/abs/2504.19254)
11+
<p align="center">
12+
<a href="https://github.com/cvs-health/uqlm/actions">
13+
<img src="https://github.com/cvs-health/uqlm/actions/workflows/ci.yaml/badge.svg" alt="Build Status">
14+
</a>
15+
16+
<a href="https://pypi.org/project/uqlm/">
17+
<img src="https://badge.fury.io/py/uqlm.svg" alt="PyPI version">
18+
</a>
19+
20+
<a href="https://cvs-health.github.io/uqlm/latest/index.html">
21+
<img src="https://img.shields.io/badge/docs-latest-blue.svg" alt="Documentation Status">
22+
</a>
23+
<a href="https://pypi.org/project/uqlm/">
24+
<img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python Versions">
25+
</a>
26+
<a href="https://opensource.org/licenses/Apache-2.0">
27+
<img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License">
28+
</a>
29+
<a href="https://github.com/astral-sh/uv">
30+
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json" alt="uv">
31+
</a>
32+
<a href="https://github.com/astral-sh/ruff">
33+
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json" alt="Ruff">
34+
</a>
35+
</p>
36+
<p align="center">
37+
<a href="https://arxiv.org/pdf/2507.06196">
38+
<img src="https://img.shields.io/badge/JMLR-Published-blue?style=flat" alt="JMLR Publication">
39+
</a>
40+
<a href="https://openreview.net/pdf?id=WOFspd4lq5">
41+
<img src="https://img.shields.io/badge/TMLR-Published-blue?style=flat" alt="TMLR Publication">
42+
</a>
43+
<a href="https://arxiv.org/abs/2602.17431">
44+
<img src="https://img.shields.io/badge/arXiv-2602.17431-B31B1B.svg" alt="arXiv">
45+
</a>
46+
</p>
1747

1848
UQLM is a Python library for Large Language Model (LLM) hallucination detection using state-of-the-art uncertainty quantification techniques.
1949

@@ -310,21 +340,22 @@ Each notebook includes detailed explanations and code samples that you can adapt
310340

311341

312342
## Citation
313-
A technical description of the `uqlm` scorers and extensive experimental results are presented in **[this paper](https://arxiv.org/abs/2504.19254)**. If you use our framework or toolkit, please cite:
343+
A technical description of the `uqlm` scorers and extensive experimental results are presented in **[this paper](https://openreview.net/pdf?id=WOFspd4lq5)**, published in **Transactions on Machine Learning Research (TMLR)**. If you use our framework or toolkit, please cite:
314344

315345
```bibtex
316-
@misc{bouchard2025uncertaintyquantificationlanguagemodels,
317-
title={Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorers},
318-
author={Dylan Bouchard and Mohit Singh Chauhan},
319-
year={2025},
320-
eprint={2504.19254},
321-
archivePrefix={arXiv},
322-
primaryClass={cs.CL},
323-
url={https://arxiv.org/abs/2504.19254},
346+
@article{
347+
bouchard2025uncertainty,
348+
title={Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, {LLM} Judge, and Ensemble Scorers},
349+
author={Dylan Bouchard and Mohit Singh Chauhan},
350+
journal={Transactions on Machine Learning Research},
351+
issn={2835-8856},
352+
year={2025},
353+
url={https://openreview.net/forum?id=WOFspd4lq5},
354+
note={}
324355
}
325356
```
326357

327-
The `uqlm` software package is described in this **[this paper](https://arxiv.org/abs/2507.06196)**. If you use the software, please cite:
358+
The `uqlm` software package is described in this **[this paper](https://arxiv.org/abs/2507.06196)**, published in the **Journal of Machine Learning Research (JMLR)**. If you use the software, please cite:
328359

329360
```bibtex
330361
@misc{bouchard2025uqlmpythonpackageuncertainty,
@@ -337,3 +368,16 @@ The `uqlm` software package is described in this **[this paper](https://arxiv.or
337368
url={https://arxiv.org/abs/2507.06196},
338369
}
339370
```
371+
372+
The long-text methods and experiment results are described in **this paper**, available as a preprint on arXiv. To cite:
373+
```bibtex
374+
@misc{bouchard2026finegraineduncertaintyquantificationlongform,
375+
title={Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Study},
376+
author={Dylan Bouchard and Mohit Singh Chauhan and Viren Bajaj and David Skarbrevik},
377+
year={2026},
378+
eprint={2602.17431},
379+
archivePrefix={arXiv},
380+
primaryClass={cs.CL},
381+
url={https://arxiv.org/abs/2602.17431},
382+
}
383+
```

‎assets/README_PYPI.md‎

Lines changed: 38 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,44 @@
33
</p>
44

55

6-
# uqlm: Uncertainty Quantification for Language Models
6+
<h1 align="center">uqlm: Uncertainty Quantification for Language Models</h1>
77

8-
[![Build Status](https://github.com/cvs-health/uqlm/actions/workflows/ci.yaml/badge.svg)](https://github.com/cvs-health/uqlm/actions)
9-
[![PyPI version](https://badge.fury.io/py/uqlm.svg)](https://pypi.org/project/uqlm/)
10-
[![Documentation Status](https://img.shields.io/badge/docs-latest-blue.svg)](https://cvs-health.github.io/uqlm/latest/index.html)
11-
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
12-
[![](https://img.shields.io/badge/arXiv-2504.19254-B31B1B.svg)](https://arxiv.org/abs/2504.19254)
8+
<p align="center">
9+
<a href="https://github.com/cvs-health/uqlm/actions">
10+
<img src="https://github.com/cvs-health/uqlm/actions/workflows/ci.yaml/badge.svg" alt="Build Status">
11+
</a>
12+
13+
<a href="https://pypi.org/project/uqlm/">
14+
<img src="https://badge.fury.io/py/uqlm.svg" alt="PyPI version">
15+
</a>
16+
17+
<a href="https://cvs-health.github.io/uqlm/latest/index.html">
18+
<img src="https://img.shields.io/badge/docs-latest-blue.svg" alt="Documentation Status">
19+
</a>
20+
<a href="https://pypi.org/project/uqlm/">
21+
<img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python Versions">
22+
</a>
23+
<a href="https://opensource.org/licenses/Apache-2.0">
24+
<img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License">
25+
</a>
26+
<a href="https://github.com/astral-sh/uv">
27+
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json" alt="uv">
28+
</a>
29+
<a href="https://github.com/astral-sh/ruff">
30+
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json" alt="Ruff">
31+
</a>
32+
</p>
33+
<p align="center">
34+
<a href="https://arxiv.org/pdf/2507.06196">
35+
<img src="https://img.shields.io/badge/JMLR-Published-blue?style=flat" alt="JMLR Publication">
36+
</a>
37+
<a href="https://openreview.net/pdf?id=WOFspd4lq5">
38+
<img src="https://img.shields.io/badge/TMLR-Published-blue?style=flat" alt="TMLR Publication">
39+
</a>
40+
<a href="https://arxiv.org/abs/2602.17431">
41+
<img src="https://img.shields.io/badge/arXiv-2602.17431-B31B1B.svg" alt="arXiv">
42+
</a>
43+
</p>
1344

1445
UQLM is a Python library for Large Language Model (LLM) hallucination detection using state-of-the-art uncertainty quantification techniques.
1546

@@ -307,4 +338,4 @@ The `uqlm` software package is described in this **[this paper](https://arxiv.or
307338
primaryClass={cs.CL},
308339
url={https://arxiv.org/abs/2507.06196},
309340
}
310-
```
341+
```

‎docs/source/contribute.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -70,8 +70,8 @@ Development Setup
7070
1. Clone the repository: ``git clone https://github.aetna.com/analytics-org/uqlm.git``
7171
2. Navigate to the project directory: ``cd uqlm``
7272
3. Create and activate a virtual environment (using ``venv`` or ``conda``)
73-
4. Install poetry (if you don't already have it): ``pip install poetry``
74-
5. Install uqlm with dev dependencies: ``poetry install --with dev``
73+
4. Install uv (if you don't already have it): ``pip install uv`` or ``curl -LsSf https://astral.sh/uv/install.sh | sh``
74+
5. Install uqlm with dev dependencies: ``uv sync --group dev``
7575
6. Install our pre-commit hooks to ensure code style compliance: ``pre-commit install``
7676
7. Run tests to ensure everything is working: ``pre-commit run --all-files```
7777

0 commit comments

Comments
 (0)