Documentation for use with mini-swe-agent

[RicardoDominguez]

Hi,

It seems that mini-swe-agent is now recommended instead of swe-agent for performing fine-tuning experiments. It would be great if the already very helpful SWE-smith tutorials could be extended to include mini-swe-agent.

Thank you for the great work!

[ofirpress]

Might take some time to get the docs updated but @klieret might have some thoughts in the meanwhile

[klieret]

Hi @RicardoDominguez , mini has a swe-bench mode that you should be able to use to run on any swe-bench style dataset. https://mini-swe-agent.com/latest/usage/swebench/

However, I think there might be some git checkout like steps that swe-smith needs, so some modding might be necessary. Hopefully we'll get to update that soon

[john-b-yang]

Just gave this a shot, things work!

$ mini-extra swebench --subset smith --filter oauthlib__oauthlib.1fd52536.combine_file__09vlzwgc --split train
👋 This is mini-swe-agent version 1.9.1.
Your config is stored in '/home/john-b-yang/.config/mini-swe-agent/.env'
/opt/miniconda3/envs/mini/lib/python3.10/site-packages/click/core.py:1193: UserWarning: The parameter -c is used more than once. Remove its duplicate as parameters should be unique.
  parser = self.make_parser(ctx)
/opt/miniconda3/envs/mini/lib/python3.10/site-packages/click/core.py:1186: UserWarning: The parameter -c is used more than once. Remove its duplicate as parameters should be unique.
  self.parse_args(ctx, args)
minisweagent: INFO: Results will be saved to .
Logging to 'minisweagent.log'
minisweagent: INFO: Loading dataset SWE-bench/SWE-smith, split train...
minisweagent: INFO: Instance filter: 59136 -> 1 instances
minisweagent: INFO: Running on 1 instances...
minisweagent.environment: DEBUG: Starting container with command: docker run -d --name minisweagent-85c4d69d -w /testbed jyangballin/swesmith.x86_64.oauthlib_1776_oauthlib.1fd52536 sleep 2h
minisweagent.environment: INFO: Started container minisweagent-85c4d69d with ID de9cbca2aa99fa3de63b6b4391f40bdc1f8c39c7fbe3a1c04f2c7cb1c46382bd
minisweagent: INFO: Saved trajectory to 'oauthlib__oauthlib.1fd52536.combine_file__09vlzwgc/oauthlib__oauthlib.1fd52536.combine_file__09vlzwgc.traj.json'
┏━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Exit Status ┃ Count ┃ Most recent instances                                   ┃
┡━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Submitted   │     1 │ oauthlib__oauthlib.1fd52536.combine_file__09vlzwgc      │
└─────────────┴───────┴─────────────────────────────────────────────────────────┘
  Overall Progress ($0.12) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/1 100% 0:02:48 eta: 0:00:00

So it's already possible to run mini-SWE-agent on SWE-smith! And it produces a folder oauthlib__oauthlib.1fd52536.combine_file__09vlzwgc, that contains the corresponding trajectory generated from attempting to solve this task.

But I will keep this thread open for a bit - are there ways we can make mini work better with SWE-agent?

[klieret]

Oh that's great, I assume it just works because it can use the docker image, right?

I think also based on this SWE-agent/mini-swe-agent#452 (comment) here, we could give the mini swebench config a run: env_startup_command field that's templated with the swe-bench instance variables. That way we could put in specific behavior

[john-b-yang]

Oh this is awesome, i see, yeah in case different instances require different kinds of setup procedures? (e.g. checkout for smith vs. git branch stuff for SWE-bench?)

[RicardoDominguez]

Great, thanks! @john-b-yang It would be helpful to include an "official" mini-swe-agent config file for SWE-bench/SWE-agent-LM-32B, akin to swesmith_gen_gpt.yaml.

[john-b-yang]

Ah ok yeah definitely sounds good + makes sense!

I think I'll add something here @klieret?

I'll also add some additional documentation that discusses this. Will get around to this things within the next week and ping + close here when it's done.

[klieret]

After this PR is merged, we'll also have support for startup commands for mini-extra swebench and mini-extra swebench-single.
The startup command can be templated with task instance variables, so that should allow for custom checkout and git branch things.

And yes, configs/extra sounds like a great place to include a config :)

[RicardoDominguez]

This is the startup command that I am using for now

run:
  env_startup_command: |
    set -euo pipefail
    REPO=/testbed
    git -C "$REPO" checkout -f {{ instance_id }}
    # prevent fetching or switching to remote branches
    git -C "$REPO" remote remove origin || true
    # delete all other local branches
    for b in $(git -C "$REPO" for-each-ref --format='%(refname:short)' refs/heads/); do
      [ "$b" = "{{ instance_id }}" ] || git -C "$REPO" branch -D "$b"
    done

[john-b-yang]

Just a ping here - @klieret and I got busy with some other duties, but will get around to adding docs for this soon!