[Enhancement]: Update documentation with example of `DockerModelRunner` usage

[scottsteen]

Module

Core

Proposal

I'm investigating using DockerModelRunner to enable integration testing of usage of an LLM. The documentation explains how to pull the image, but does not go on to explain the API or how to use the container after it is started.

I'm also unclear on the differences between getBaseEndpoint() & getOpenAIEndpoint(). There are no Javadocs for either.

For example, if I want to invoke OpenAI's Responses API, how would/can I do that with DockerModelRunner? Is this the right line of thinking? If not, what is? Is the underlying model important for using the OpenAI APIs?

I'm new to developing LLM-based applications, so any help is welcomed.

This request isn't to help with how to write the assertions/evals.

[scottsteen]

Having looked through the tests, I wondered if the Responses API was supported, which it appears not. I'm not sure with whom that support would live - I imagine it's not Testcontainers, so not asking for that.

The Chat Completions API is supported, so have gone with that.

To answer my own question, an example would look something like this:

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.DockerModelRunnerContainer;

class ModelRunnerTest {

  static DockerModelRunnerContainer modelRunner = new DockerModelRunnerContainer("alpine/socat:1.7.4.3-r0")
    .withModel("ai/smollm2");

  static {
    modelRunner.start();
  }

  @Test
  public void describesCheese() throws Exception {
    try (var client = HttpClient.newHttpClient()) {
      var request = HttpRequest.newBuilder()
        .uri(URI.create(modelRunner.getOpenAIEndpoint() + "/v1/chat/completions"))
        .header("Content-Type", "application/json")
        .POST(HttpRequest.BodyPublishers.ofString("""
          {
            "model": "ai/smollm2",
            "messages": [
              {
                "role": "user",
                "content": "What is cheese?"
              }
            ]
          }"""))
        .build();

      var response = client.send(request, HttpResponse.BodyHandlers.ofString());

      System.out.println(response.body());
    }
  }
}

[linghengqian]

• This sounds like it belongs to what https://docs.docker.com/ai/model-runner/#what-api-endpoints-are-available and https://docs.docker.com/ai/model-runner/#how-do-i-interact-through-the-openai-api explain?
• I am relatively unfamiliar with docker-model-plugin. On the one hand, the source code and binary of docker-model-plugin are indeed under apache v2 license. On the other hand, docker-model-plugin is still in beta state. But the problem is that building docker-model-plugin requires docker's closed source software, refer to Can't build due to absent github.com/docker/pinata repository docker/model-cli#11. docker-model-plugin currently only provides binary on linux, and I can't test this module through windows. 🫠