Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -386,6 +386,33 @@ for model in models:
print(model)
```

### Batch Inference

The batch API allows you to submit larger inference jobs for completion with a 24 hour turn-around time, below is an example. To learn more refer to the [docs here](https://docs.together.ai/docs/batch-inference).

```python
from together import Together

client = Together()

# Upload the batch file
batch_file = client.files.upload(file="simpleqa_batch_student.jsonl", purpose="batch-api")

# Create the batch job
batch = client.batches.create_batch(file_id=batch_file.id, endpoint="/v1/chat/completions")

# Monitor the batch status
batch_stat = client.batches.get_batch(batch.id)

# List all batches - contains other batches as well
client.batches.list_batches()

# Download the file content if job completed
if batch_stat.status == 'COMPLETED':
output_response = client.files.retrieve_content(id=batch_stat.output_file_id,
output="simpleqa_v3_output.jsonl")
```

## Usage – CLI

### Chat Completions
Expand Down