feat: airtbench ai agent code #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

GangGreenTemperTatum merged 14 commits into main from ads/eng-2127-feat-create-airtbench-code-repo-and-merge-in-example-agents

Jun 5, 2025

Collaborator

GangGreenTemperTatum commented Jun 4, 2025 •

edited

Loading

AIRTBench AI Agent Code

Note: still pending team feedback on walkthrough agent code docs (related PR) for best location to deploy this to compliment the agent code

Key Changes:

includes AIRTBench AI agent code
sufficient README docs to get started
simple example notebook for extracting metrics from the eval

Added:

AIRTBench AI agent code
sufficient README docs to get started
simple example notebook for extracting metrics from the eval

Changed:

NA

Removed:

NA


          feat: airtbench ai agent code

9a5ae64

GangGreenTemperTatum requested a review from moohax

June 4, 2025 14:04

GangGreenTemperTatum self-assigned this

linear bot commented Jun 4, 2025

ENG-2127 feat: create `AIRTBench-Code` repo and merge in `example-agents` branch

ads said:

could i ask for a peer-reviewer of my example-agent (local branch) code and prod-docs (remote branch) please? (maybe ill share a zip of my local repo for local branch review since its non-public rn)
it's for AIRTBench and wanting to ensure it's ready to drop at the same time as the paper - believe everything should be good to roll from my end and tested
please let me know and i can throw up tasks and link everything in a thread here .. tyia! 🙂

GangGreenTemperTatum added 2 commits

June 4, 2025 10:07


          fix: add groq generator for pr janitor

04c9dc9


          Merge branch 'main' into ads/eng-2127-feat-create-airtbench-code-repo…

70a6402

…-and-merge-in-example-agents

dreadnode-renovate-bot bot added area/pre-commit area/python type/docs type/core labels

moohax requested a review from rdheekonda

June 4, 2025 21:49

rdheekonda reviewed

View reviewed changes

.env.example Outdated Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

.env.example Outdated Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

airtbench/challenges/bear4.ipynb Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

airtbench/challenges/bear4.ipynb Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

airtbench/challenges/bear4.ipynb Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

airtbench/challenges/bear4.ipynb Outdated Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

airtbench/challenges/bear4.ipynb Outdated Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

airtbench/challenges/brig1.ipynb Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

airtbench/challenges/brig1.ipynb Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

airtbench/challenges/brig2.ipynb Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

airtbench/container.py Show resolved Hide resolved


          chore: add env example placeholders

2d0fbcb

rdheekonda reviewed

View reviewed changes

airtbench/container.py Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

airtbench/container/Dockerfile Outdated Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

airtbench/container/Dockerfile Show resolved Hide resolved


          fix: rm dup torchvision

38ef6c1

rdheekonda reviewed

View reviewed changes

notebooks/airtbench_api_analysis.ipynb Outdated Show resolved Hide resolved

GangGreenTemperTatum added 2 commits

June 4, 2025 19:47


          fix: rm callisto refs

cfaf123


          fix: rm callisto refs in notebook

7902f9d

rdheekonda reviewed

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved


          chore: rm duckdb from toml

f41a7cd

rdheekonda reviewed

View reviewed changes

airtbench/main.py Outdated Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

airtbench/main.py Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

airtbench/main.py Show resolved Hide resolved

rdheekonda reviewed

View reviewed changes

airtbench/main.py Show resolved Hide resolved

GangGreenTemperTatum added 6 commits

June 4, 2025 21:00


          chore: bear4 notebook metadata

53a1e77


          chore: standardize single env var for challenges n strikes

7e1a771


          chore: bear4 queries feedback

51e77b6


          chore: neutral terminology in bear4 notebook

e245e3a


          chore: add max retries to flag submission

5e919bd


          fix: platform hyperlink

e68954f

rdheekonda approved these changes

View reviewed changes

GangGreenTemperTatum merged commit 02d0fd6 into main

4 checks passed

GangGreenTemperTatum deleted the ads/eng-2127-feat-create-airtbench-code-repo-and-merge-in-example-agents branch

June 5, 2025 15:01

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/pre-commit area/python type/core type/docs