Rest API support #49

mikepapadim · 2025-09-08T12:22:18Z

No description provided.

Copilot

Pull Request Overview

This PR adds REST API support to the existing TornadoVM LLM application, enabling it to function as a web service alongside the existing command-line interface. The implementation provides OpenAI-compatible endpoints for text completion with both regular and streaming responses.

Key changes:

Added Spring Boot web framework integration with REST API endpoints
Refactored sampler creation logic from LlamaApp to Sampler interface for reusability
Created service layer architecture for model initialization and text generation

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
ModelLoader.java	Added static loadModel method with AOT support for service initialization
Sampler.java	Added static factory methods for sampler creation previously in LlamaApp
TokenizerService.java	New service for text encoding/decoding operations
ModelInitializationService.java	New service for initializing model, options, and sampler on startup
LLMService.java	New service handling text completion generation (regular and streaming)
CompletionResponse.java	Response model class for OpenAI-compatible completion API
CompletionRequest.java	Request model class for OpenAI-compatible completion API
CompletionController.java	REST controller providing /v1/completions endpoints
ModelConfiguration.java	Spring configuration for exposing model and options as beans
LLMApiApplication.java	Spring Boot application entry point for API service mode
LlamaApp.java	Removed sampler and model loading methods (moved to respective classes)
pom.xml	Added Spring Boot and Jackson dependencies
llama-tornado	Added --service flag and updated argument handling for API mode

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-16T15:07:09Z

src/main/java/org/beehive/gpullama3/model/loader/ModelLoader.java

+     //     * Loads the language model based on the given options.
+     //     * <p>
+     //     * If Ahead-of-Time (AOT) mode is enabled, attempts to use a pre-loaded compiled model. Otherwise, loads the model from the specified path using the model loader.
+     //     * </p>
+     //     *
+     //     * @param options
+     //     *         the parsed CLI options containing model path and max token limit
+     //     * @return the loaded {@link Model} instance
+     //     * @throws IOException
+     //     *         if the model fails to load
+     //     * @throws IllegalStateException
+     //     *         if AOT loading is enabled but the preloaded model is unavailable
+     //     */


The comment block uses incorrect comment syntax with '//' instead of '*' for Javadoc. This should be standard Javadoc format to be properly processed by documentation tools.

Suggested change

// * Loads the language model based on the given options.

// * <p>

// * If Ahead-of-Time (AOT) mode is enabled, attempts to use a pre-loaded compiled model. Otherwise, loads the model from the specified path using the model loader.

// * </p>

// *

// * @param options

// * the parsed CLI options containing model path and max token limit

// * @return the loaded {@link Model} instance

// * @throws IOException

// * if the model fails to load

// * @throws IllegalStateException

// * if AOT loading is enabled but the preloaded model is unavailable

// */

* Loads the language model based on the given options.

* <p>

* If Ahead-of-Time (AOT) mode is enabled, attempts to use a pre-loaded compiled model. Otherwise, loads the model from the specified path using the model loader.

* </p>

*

* @param options

* the parsed CLI options containing model path and max token limit

* @return the loaded {@link Model} instance

* @throws IOException

* if the model fails to load

* @throws IllegalStateException

* if AOT loading is enabled but the preloaded model is unavailable

*/

Copilot · 2025-09-16T15:07:09Z

src/main/java/org/beehive/gpullama3/api/service/TokenizerService.java

+//    public String decode(int token) {
+//        Model model = initService.getModel();
+//        // Convenience method for single token decoding
+//        return model.tokenizer().decode(token);
+//    }


Remove commented-out code. If this functionality might be needed later, consider implementing it or documenting why it's disabled.

Suggested change

// public String decode(int token) {

// Model model = initService.getModel();

// // Convenience method for single token decoding

// return model.tokenizer().decode(token);

// }

Copilot · 2025-09-16T15:07:10Z

src/main/java/org/beehive/gpullama3/api/service/LLMService.java

+                }
+
+                // Create custom sampler with request-specific parameters
+                //Sampler sampler = initService.createCustomSampler(temperature, topP, System.currentTimeMillis());


Remove commented-out code that references a non-existent method. This creates confusion about the intended functionality.

Suggested change

//Sampler sampler = initService.createCustomSampler(temperature, topP, System.currentTimeMillis());

Copilot · 2025-09-16T15:07:10Z

src/main/java/org/beehive/gpullama3/api/service/LLMService.java

+                    }
+                }
+
+                //Sampler sampler = initService.createCustomSampler(temperature, topP, System.currentTimeMillis());


Remove commented-out code that references a non-existent method. This creates confusion about the intended functionality.

Suggested change

//Sampler sampler = initService.createCustomSampler(temperature, topP, System.currentTimeMillis());

Copilot

Pull Request Overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-18T09:21:36Z

src/main/java/org/beehive/gpullama3/model/loader/ModelLoader.java

+            return ModelLoader.loadModel(options.modelPath(), options.maxTokens(), true);
+        }
+
    public static Model loadModel(Path ggufPath, int contextLength, boolean loadWeights) throws IOException {


This line calls ModelLoader.loadModel() from within the ModelLoader class itself, but the method name is ambiguous. Consider renaming this overloaded method to avoid confusion, such as 'loadModelFromPath' or similar.

Suggested change

return ModelLoader.loadModel(options.modelPath(), options.maxTokens(), true);

}

public static Model loadModel(Path ggufPath, int contextLength, boolean loadWeights) throws IOException {

return ModelLoader.loadModelFromPath(options.modelPath(), options.maxTokens(), true);

}

public static Model loadModelFromPath(Path ggufPath, int contextLength, boolean loadWeights) throws IOException {

Copilot · 2025-09-18T09:21:36Z

src/main/java/org/beehive/gpullama3/api/service/LLMService.java

+            // Step 2: Load model weights
+            System.out.println("\nStep 2: Loading model...");
+            System.out.println("Loading model from: " + options.modelPath());
+            model = ModelLoader.loadModel(options.modelPath(), options.maxTokens(), true);


The service is calling the static loadModel method directly instead of using the new loadModel(Options) method that was added to ModelLoader. This bypasses the AOT functionality. Consider using ModelLoader.loadModel(options) instead.

Suggested change

model = ModelLoader.loadModel(options.modelPath(), options.maxTokens(), true);

model = ModelLoader.loadModel(options);

Copilot

Pull Request Overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-19T11:28:51Z

src/main/java/org/beehive/gpullama3/api/service/LLMService.java

+            if (!generatedTokens.isEmpty() && stopTokens.contains(generatedTokens.getLast())) {
+                generatedTokens.removeLast();


The getLast() and removeLast() methods are Java 21+ features. This code will fail to compile on Java versions prior to 21, potentially causing compatibility issues if the project needs to support older Java versions.

Suggested change

if (!generatedTokens.isEmpty() && stopTokens.contains(generatedTokens.getLast())) {

generatedTokens.removeLast();

if (!generatedTokens.isEmpty() && stopTokens.contains(generatedTokens.get(generatedTokens.size() - 1))) {

generatedTokens.remove(generatedTokens.size() - 1);

Copilot · 2025-09-19T11:28:51Z

src/main/java/org/beehive/gpullama3/api/model/CompletionResponse.java

+        this.id = "cmpl-" + System.currentTimeMillis();
+        this.created = System.currentTimeMillis() / 1000;


Using System.currentTimeMillis() twice in quick succession may result in different timestamps for id and created. Consider calling it once and storing the result in a variable to ensure consistency.

Suggested change

this.id = "cmpl-" + System.currentTimeMillis();

this.created = System.currentTimeMillis() / 1000;

long nowMillis = System.currentTimeMillis();

this.id = "cmpl-" + nowMillis;

this.created = nowMillis / 1000;

Copilot · 2025-09-19T11:28:51Z

src/main/java/org/beehive/gpullama3/api/service/LLMService.java

+                                if (model.tokenizer().shouldDisplayToken(token)) {
+                                    String tokenText = model.tokenizer().decode(List.of(token));
+                                    emitter.send(SseEmitter.event().data(tokenText));
+                                    //emitter.send(SseEmitter.event().comment("flush"));


This commented-out code should either be removed if no longer needed or properly implemented if it serves a purpose for flushing the SSE stream.

Suggested change

//emitter.send(SseEmitter.event().comment("flush"));

mikepapadim marked this pull request as ready for review September 16, 2025 15:06

mikepapadim requested review from Copilot and orionpapadakis September 16, 2025 15:06

mikepapadim self-assigned this Sep 16, 2025

Copilot AI reviewed Sep 16, 2025

View reviewed changes

mikepapadim requested a review from Copilot September 18, 2025 09:20

Copilot AI reviewed Sep 18, 2025

View reviewed changes

orionpapadakis force-pushed the feat/rest-api branch from 2b2ed4d to ba97fe9 Compare September 18, 2025 18:14

orionpapadakis added 8 commits September 19, 2025 14:25

Initial commit for GPULlama3.java REST API support

e50b1c0

[WIP] Simplify service logic with ChatController

a4e103d

Update llama-tornado python script for service

9a1a475

Add serviceMode Options field and parseServiceOptions method

3eabd75

[WIP] Update service logic

60c67e6

[WIP] Update service logic

560e641

[WIP] Start adding logic for tornado path in service

5143753

Minor changes

6b76b08

orionpapadakis force-pushed the feat/rest-api branch from ba97fe9 to 6b76b08 Compare September 19, 2025 11:27

mikepapadim requested a review from Copilot September 19, 2025 11:27

Copilot AI reviewed Sep 19, 2025

View reviewed changes

orionpapadakis added 4 commits September 19, 2025 14:36

Set tornado default to false for service

17a6275

Use a Consumer on a generated token

50d7536

Add get methods

c557b5e

Add GPU path to Rest API

93d0c93

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rest API support #49

Rest API support #49

Uh oh!

mikepapadim commented Sep 8, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 16, 2025

Uh oh!

Copilot AI Sep 16, 2025

Uh oh!

Copilot AI Sep 16, 2025

Uh oh!

Copilot AI Sep 16, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 18, 2025

Uh oh!

Copilot AI Sep 18, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 19, 2025

Uh oh!

Copilot AI Sep 19, 2025

Uh oh!

Copilot AI Sep 19, 2025

Uh oh!

Uh oh!

	model = ModelLoader.loadModel(options.modelPath(), options.maxTokens(), true);
	model = ModelLoader.loadModel(options);

		if (!generatedTokens.isEmpty() && stopTokens.contains(generatedTokens.getLast())) {
		generatedTokens.removeLast();

		this.id = "cmpl-" + System.currentTimeMillis();
		this.created = System.currentTimeMillis() / 1000;

-        this.id = "cmpl-" + System.currentTimeMillis();
-        this.created = System.currentTimeMillis() / 1000;
+        long nowMillis = System.currentTimeMillis();
+        this.id = "cmpl-" + nowMillis;
+        this.created = nowMillis / 1000;

Rest API support #49

Are you sure you want to change the base?

Rest API support #49

Uh oh!

Conversation

mikepapadim commented Sep 8, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!