TogetherSolver (#1502)

This PR contributes a `TogetherSolver` class, a solver for using models
served by the [Together AI
API](https://docs.together.ai/docs/quickstart)

Because [Together supports the OpenAI python
sdk](https://docs.together.ai/docs/openai-api-compatibility), we simply
create a subclass of the `OpenAISolver`, overriding some functionality.
There is therefore some refactoring of the `OpenAISolver` included in
this PR to facilitate this code sharing.

At the moment, we support the models specified in
`evals/registry/solvers/together.yaml`, but in principle most models
offered from the Together AI API can easily be added

Notes:

- logit biasing not supported by the Together API due to a lack of a
unified tokenizer a la [tiktoken](https://github.com/openai/tiktoken)
from openai
- For the same reason, checking for context length limits not supported

Co-authored-by: Chan Jun Shern <[email protected]>
Co-authored-by: Ian McKenzie <[email protected]>

Author: 3 people

evals/registry/solvers/together.yaml

@@ -0,0 +1,92 @@
+# --- Direct Solvers --- 
+generation/direct/llama-2-13b-chat:
+  class: evals.solvers.together_solver:TogetherSolver
+  args:
+    completion_fn_options:
+      model: meta-llama/Llama-2-13b-chat-hf
+      extra_options:
+        temperature: 1
+        max_tokens: 512
+    postprocessors: &postprocessors
+      - evals.solvers.postprocessors.postprocessors:Strip
+
+generation/direct/llama-2-70b-chat:
+  class: evals.solvers.together_solver:TogetherSolver
+  args:
+    completion_fn_options:
+      model: meta-llama/Llama-2-70b-chat-hf
+      extra_options:
+        temperature: 1
+        max_tokens: 512
+    postprocessors: *postprocessors
+
+generation/direct/mixtral-8x7b-instruct:
+  class: evals.solvers.together_solver:TogetherSolver
+  args:
+    completion_fn_options:
+      model: mistralai/Mixtral-8x7B-Instruct-v0.1
+      extra_options:
+        temperature: 1
+        max_tokens: 512
+    postprocessors: *postprocessors
+# --- COT Solvers ---
+
+generation/cot/llama-2-13b-chat:
+  class: evals.solvers.nested.cot_solver:CoTSolver
+  args:
+    cot_solver:
+      class: evals.solvers.together_solver:TogetherSolver
+      args:
+        completion_fn_options:
+          model: meta-llama/Llama-2-13b-chat-hf
+          extra_options:
+            temperature: 1
+            max_tokens: 512
+    extract_solver:
+      class: evals.solvers.together_solver:TogetherSolver
+      args:
+        completion_fn_options:
+          model: meta-llama/Llama-2-13b-chat-hf
+          extra_options:
+            temperature: 1
+            max_tokens: 512
+
+generation/cot/llama-2-70b-chat:
+  class: evals.solvers.nested.cot_solver:CoTSolver
+  args:
+    cot_solver:
+      class: evals.solvers.together_solver:TogetherSolver
+      args:
+        completion_fn_options:
+          model: meta-llama/Llama-2-70b-chat-hf
+          extra_options:
+            temperature: 1
+            max_tokens: 512
+    extract_solver:
+      class: evals.solvers.together_solver:TogetherSolver
+      args:
+        completion_fn_options:
+          model: meta-llama/Llama-2-70b-chat-hf
+          extra_options:
+            temperature: 1
+            max_tokens: 512
+
+generation/cot/mixtral-8x7b-instruct:
+  class: evals.solvers.nested.cot_solver:CoTSolver
+  args:
+    cot_solver:
+      class: evals.solvers.together_solver:TogetherSolver
+      args:
+        completion_fn_options:
+          model: mistralai/Mixtral-8x7B-Instruct-v0.1
+          extra_options:
+            temperature: 1
+            max_tokens: 512
+    extract_solver:
+      class: evals.solvers.together_solver:TogetherSolver
+      args:
+        completion_fn_options:
+          model: mistralai/Mixtral-8x7B-Instruct-v0.1
+          extra_options:
+            temperature: 1
+            max_tokens: 512