Fake AI provider

[wkordalski]

Context

Related issue: #1656

We would like to test the integration of roospawn with Roo-Code.
Testing integration using existing AI API providers is not an option, because they give us a very limited control over the AI provider behavior. They make impossible to test edge cases when the order of events is important or cases when the provider fails or timeouts.

The partial solution could be the Human Relay provider, however controlling it from test code is quite difficult.

I would like to suggest another provider (lets call it "Fake AI" provider) that gets an AI implementation object via configuration (RooCodeAPI.setConfiguration()) and proxies all the calls to it.

Implementation

Add FakeAI provider which proxies all calls to its methods to an "implementation object" that is passed via configuration.

Screenshots

None

How to Test

Example usage of FakeAI provider. We will use similar code in our integration tests.

// ModelInfo, ApiStream, ApiStreamChunk, ApiStreamTextChunk, ApiStreamReasoningChunk, ApiStreamUsageChunk
// copied from relevant files in Roo-Code sources.

class FakeAi {
    messagesCount: number = 0;
    async *createMessage(systemPrompt: string, messages: Anthropic.Messages.MessageParam[]) {
        this.messagesCount++;

        // Wait for checkpoint creation — otherwise an error is shown
        await new Promise(resolve => setTimeout(resolve, 1000));
        yield { type: "text", text: "Hello, world!" };
        yield { type: "text", text: "<attempt_completion><result>Results here</result></attempt_completion>" };
    }

    getModel(): { id: string; info: ModelInfo } {
        return { id: 'fake-ai', info: { contextWindow: 10000, supportsPromptCache: false }};
    }
    async countTokens(content: Array<Anthropic.Messages.ContentBlockParam>): Promise<number> {
        return 0;
    }
}

// and in test...

const roo_code_api = roo_code_extension.exports;
const fakeAi = new FakeAi();
roo_code_api.setConfiguration({ 'apiProvider': 'fake-ai', 'fakeAi': fakeAi });

const task_id = await roo_code_api.startNewTask("My task");

await new Promise<void>(resolve => roo_code_api.on('message', ({taskId, message}) => {
    if (taskId === task_id && !message.partial && message.say === 'completion_result') {
        resolve();
    }
}));

console.log(fakeAi.messagesCount === 1);

Get in Touch

Discord handle: wkordalski

---

Important

Add FakeAI provider to enable testing of AI integrations with configurable behavior.

• Behavior:
  • Adds FakeAIHandler in fake-provider.ts to proxy calls to a configurable implementation object.
  • Updates buildApiHandler() in index.ts to include fake-ai as a provider option.
• Configuration:
  • Adds fakeAi to ApiHandlerOptions and ApiConfiguration in api.ts.
  • Updates GLOBAL_STATE_KEYS and API_CONFIG_KEYS to include fakeAi.
• UI:
  • Adds Fake AI option to provider dropdown in ApiOptions.tsx.

^{This description was created by for f9190ecc6def0d23546c25f61e019ec8840ec67a. It will automatically update as commits are pushed.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 5cba8ee

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[ellipsis-dev]

New provider 'fake-ai' added to the providers array. However, there is no specific UI handling for this provider in normalizeApiConfiguration. Consider adding an explicit case to handle 'fake-ai' configuration if special behaviour is required.

[cte]

Should we hide this in the settings UI? I'm assuming this will only be used programmatically.

[franekp]

I think it is better to hide it. My only concern would be if the fake-ai provider is set via RooCodeAPI.setConfiguration, and later the settings view is opened as part of the e2e test, wouldn't this lead to some inconsistent state and possibly the setting view "normalizing" and overwriting the config value? Although, you could just say "don't open settings during e2e tests" and that would be fine for us, since we can use setConfiguration to setup settings for a test.

[cte]

I think we can deal with this if it ever comes up in the e2e tests; for now let's optimize for not causing user confusion and hide this in the settings.

[wkordalski]

Fixed.
It seems that when Fake AI is set as an API provider, in the settings view the API Provider input simply shows an empty field (no API provider is selected).
Saving the settings will clear the fakeAi object that handles API requests (and thus further requests will fail until the user sets another API provider). Showing settings view without saving does not overwrite configuration.