Start building out Aider-style benchmark infra

Author: cte

benchmark/.gitignore

@@ -0,0 +1,2 @@
+repos/*
+!repos/.gitkeep

benchmark/README.md

@@ -0,0 +1,31 @@
+Get the exercises from exercism:
+
+````sh
+gh repo clone exercism/typescript repos/typescript
+# Same idea for python, ruby, go, rust, etc.
+# gh repo clone exercism/python repos/python
+find . -name ".vscode" | xargs rm -rf```
+````
+
+Prepare an exercise:
+
+```sh
+LANG=typescript
+EXERCISE=flatten-array
+
+cp -rf repos/$LANG/exercises/practice/$EXERCISE $LANG
+sed -i 's/\bxit(/it(/g' $LANG/$EXERCISE/*.test.ts
+
+echo "# Task
+
+Complete the implementation in \`$EXERCISE.ts\`.
+
+$(cat $LANG/$EXERCISE/.docs/instructions.md)
+
+# Confirmation
+
+To confirm that your solution is correct, run the tests with \`yarn test\`.
+You might need to install the dependencies first with \`yarn install\`." > $LANG/$EXERCISE/.docs/prompt.md
+```
+
+TODO: The prompt is language dependent, so let's create a template for each language. The example above is for typescript.

benchmark/go/.gitkeep

@@ -0,0 +1,3 @@
+# Instructions Append
+
+The anagrams can be returned in any order.

benchmark/python/.gitkeep

@@ -0,0 +1,13 @@
+# Instructions
+
+Your task is to, given a target word and a set of candidate words, to find the subset of the candidates that are anagrams of the target.
+
+An anagram is a rearrangement of letters to form a new word: for example `"owns"` is an anagram of `"snow"`.
+A word is _not_ its own anagram: for example, `"stop"` is not an anagram of `"stop"`.
+
+The target and candidates are words of one or more ASCII alphabetic characters (`A`-`Z` and `a`-`z`).
+Lowercase and uppercase characters are equivalent: for example, `"PoTS"` is an anagram of `"sTOp"`, but `StoP` is not an anagram of `sTOp`.
+The anagram set is the subset of the candidate set that are anagrams of the target (in any order).
+Words in the anagram set should have the same letter case as in the candidate set.
+
+Given the target `"stone"` and candidates `"stone"`, `"tones"`, `"banana"`, `"tons"`, `"notes"`, `"Seton"`, the anagram set is `"tones"`, `"notes"`, `"Seton"`.