[PB] Trouble installing PaperBench as a dependancy

[stalkermustang]

Hello,
adding paperbench breaks the installation:

uv sync
Using CPython 3.12.11
Creating virtual environment at: .venv
   Updating https://github.com/openai/frontier-evals.git (7acfcc7f4e324d3a1058e039e5630ea5d884152c)                     × Failed to download and build `paperbench @
  │ git+https://github.com/openai/frontier-evals.git@7acfcc7f4e324d3a1058e039e5630ea5d884152c#subdirectory=project/paperbench`
  ├─▶ Git operation failed
  ╰─▶ process didn't exit successfully: `/usr/bin/git reset --hard 7acfcc7f4e324d3a1058e039e5630ea5d884152c` (exit
      status: 128)
      --- stderr
      Downloading project/paperbench/data/judge_eval/all-in-one/0/submission.tar (223 MB)
      Error downloading object: project/paperbench/data/judge_eval/all-in-one/0/submission.tar (01bfb15):
      Smudge error: Error downloading project/paperbench/data/judge_eval/all-in-one/0/submission.tar
      (01bfb15f258d7543cc4d5aba7f664ce9e97bef9e71f73fccffaa561b7640a78f): error transferring
      "01bfb15f258d7543cc4d5aba7f664ce9e97bef9e71f73fccffaa561b7640a78f": [0] remote missing object
      01bfb15f258d7543cc4d5aba7f664ce9e97bef9e71f73fccffaa561b7640a78f

      Errors logged to
      '/Users/seeall/.cache/uv/git-v0/checkouts/b6fde5aacfc5e4d8/7acfcc7/.git/lfs/logs/20251031T013910.75333.log'.
      Use `git lfs logs last` to view the log.
      error: external filter 'git-lfs filter-process' failed
      fatal: project/paperbench/data/judge_eval/all-in-one/0/submission.tar: smudge filter lfs failed

The pyproject.toml file has this to set the dependancy:

dependencies = [
    "paperbench",
]

[tool.uv.sources]
paperbench = {git = "https://github.com/openai/frontier-evals.git", subdirectory = "project/paperbench", rev = "7acfcc7f4e324d3a1058e039e5630ea5d884152c"}

Using GIT_LFS_SKIP_SMUDGE=1 helps with the issue, but it seems there's a deeper problem here that the flag merely avoids.

I asked Codex if I can resolve the issue manually, but it responded this:

Reach out to the frontier-evals maintainers (or file an issue/PR) so they either push the missing LFS object or rewrite that commit to drop the pointe

And here I am :)

[thesofakillers]

Hey @stalkermustang . Thanks for raising this, was on my bucket list of things to fix and this gave me an excuse to pay attention to it.

I've fixed this in #84. Before I merge, can you try it out? Basically try redoing what you were doing but with the following in your pyproject.toml

[tool.uv.sources]
paperbench = { git = "https://github.com/openai/frontier-evals.git", subdirectory = "project/paperbench", rev = "pb/lfs-install-fix" }

I think you need to call uv cache clean paperbench first though

[stalkermustang]

Hey @thesofakillers , thanks for the quick response!
Yep, tested with the change — works for me.

[thesofakillers]

great, merging!

[thesofakillers]

fixed via #84