Skip to content

[SWEL] Update readme #67

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
thesofakillers merged 1 commit into from
Aug 26, 2025
Merged

[SWEL] Update readme #67

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 24 additions & 9 deletions project/swelancer/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@
# SWE-Lancer

This repo contains the dataset and code for the paper "SWE-Lancer: Can Frontier
LLMs Earn \$1 Million from Real-World Freelance Software Engineering?".
This repo contains the dataset and code for the paper ["SWE-Lancer: Can Frontier
LLMs Earn \$1 Million from Real-World Freelance Software Engineering?"](https://arxiv.org/pdf/2502.12115).

> Note: as of 2025/07/17, this repo contains a subset of the original 237
> problems mentioned in the SWE-Lancer paper. Specifically, this repo has 198 tasks
> that were adjusted and verified to run successfully offline. We have dropped the
> remaining 39 problems.

## Setup

Expand Down Expand Up @@ -48,9 +53,9 @@ Note, if you do not want to skip pushing the images, then you should remove the
`--skip-push` argument and provide a `--registry` argument.

Note, we have also already pushed the images to dockerhub, which can be found
[here](https://hub.docker.com/orgs/swelancer/repositories).
[here](https://hub.docker.com/u/swelancer).

Note, to run SWE Manager tasks, you must use the "monolith" image.
Note, to run SWE Manager tasks, you must use the "monolith" image.

### Environment variables

Expand All @@ -61,8 +66,12 @@ contains template environment variables needed for the application:

```plaintext
# sample.env contents example:
OPENAI_API_KEY=
OPENROUTER_API_KEY=

USE_WEB_PROXY=false
# ... other variables
EXPENSIFY_URL=https://www.expensify.com/
NEW_EXPENSIFY_URL=https://new.expensify.com/
```

Create a new file named `.env` and copy the contents from `sample.env`, setting
Expand All @@ -73,10 +82,16 @@ the application. Usually you just need to set `OPENAI_API_KEY` or
## Running SWELancer

> Note: SWELancer is designed to run with internet disabled. At the moment,
> this is only supported on Linux systems, as it relies on `iptables`. To run
> SWELancer on MacOS, you will need to enable internet access. Simply add the
> `swelancer.disable_internet=False` argument to any of the run commands below.
> Note that running swelancer with internet enabled is not well tested.
> this is only supported on Linux systems, as it relies on `iptables`. It is
> possible to run on MacOS by adding the `swelancer.disable_internet=False`
> argument to any run commands. However, note that we have observed that some
> tasks behave abnormally when run with internet enabled, and we only consider
> rollouts run with internet disabled to be valid. Finally, while disabling
> internet should not have undesired side effects on the host machine,
> because of the nature of running iptables commands, we recommend running
> swelancer on an ephemeral VM. If you notice any weird effects, try running
> `sudo iptables -S`, and delete any rules that have the comment
> `alcatraz_block`.

We've implemented a [DummySolver](swelancer/solvers/dummy/solver.py) which can
be used to verify that the evaluation works as intended:
Expand Down