Dev

[shubhammalhotra28]

Description

Tool calling in all 4 SDK's + VLM and Diffusion model support for SWIFT SDK!

Type of Change

• Bug fix
• New feature
• Documentation update
• Refactoring

Testing

• Lint passes locally
• Added/updated tests for changes

Platform-Specific Testing (check all that apply)

Swift SDK / iOS Sample:

• Tested on iPhone (Simulator or Device)
• Tested on iPad / Tablet
• Tested on Mac (macOS target)

Kotlin SDK / Android Sample:

• Tested on Android Phone (Emulator or Device)
• Tested on Android Tablet

Flutter SDK / Flutter Sample:

• Tested on iOS
• Tested on Android

React Native SDK / React Native Sample:

• Tested on iOS
• Tested on Android

Labels

Please add the appropriate label(s):

SDKs:

• Swift SDK - Changes to Swift SDK (sdk/runanywhere-swift)
• Kotlin SDK - Changes to Kotlin SDK (sdk/runanywhere-kotlin)
• Flutter SDK - Changes to Flutter SDK (sdk/runanywhere-flutter)
• React Native SDK - Changes to React Native SDK (sdk/runanywhere-react-native)
• Commons - Changes to shared native code (sdk/runanywhere-commons)

Sample Apps:

• iOS Sample - Changes to iOS example app (examples/ios)
• Android Sample - Changes to Android example app (examples/android)
• Flutter Sample - Changes to Flutter example app (examples/flutter)
• React Native Sample - Changes to React Native example app (examples/react-native)

Checklist

• Code follows project style guidelines
• Self-review completed
• Documentation updated (if needed)

Screenshots

Attach relevant UI screenshots for changes (if applicable):

• Mobile (Phone)
• Tablet / iPad
• Desktop / Mac

Greptile Overview

Greptile Summary

This PR merges the dev branch into main, introducing three major feature additions across all SDK platforms (Swift, Kotlin, Flutter, React Native):

Major Changes

1. Tool Calling System (~3,500 lines)

• Centralized C++ implementation in runanywhere-commons as single source of truth
• Support for multiple LLM formats: DEFAULT (<tool_call>JSON</tool_call>) and LFM2 (<|tool_call_start|>[func(args)]<|tool_call_end|>)
• Comprehensive parsing with edge case handling (missing tags, unquoted keys, brace matching)
• Thin platform wrappers in Swift, Kotlin, Flutter, and React Native
• All SDKs delegate to commons layer (no duplicate parsing logic)
• Integration in all example apps with UI components for tool call visualization

2. Diffusion/Image Generation (~2,800 lines)

• New diffusion feature for text-to-image, image-to-image, and inpainting
• Backend support for CoreML (Apple) and ONNX (cross-platform)
• Comprehensive type system with tokenizer, model registry, and platform services
• Integration across all SDKs with view models and UI in example apps

3. Vision-Language Models (VLM) (~2,600 lines)

• Multimodal AI support for image understanding with LlamaCPP backend
• Image utilities with base64 decoding and stb_image integration
• Camera integration in iOS example app
• Full SDK support across platforms with FFI bridges

Additional Changes

• Added nlohmann/json v3.11.3 library (24,765 lines) for React Native JSON parsing
• Secure math expression parser in React Native example (avoids eval())
• Build system updates across all platforms
• UI components and view models in all example apps
• Package dependency updates

Architecture

• Follows established pattern: C++ commons for core logic, thin platform wrappers
• Maintains separation of concerns between SDK core and example apps
• Consistent API surface across all platform SDKs

Confidence Score: 3/5

• This PR is moderately safe to merge but requires thorough testing due to its massive scope
• Score of 3 reflects a well-architected feature addition with ~60K lines of changes across 224 files. The code quality is high with proper abstractions and consistent patterns, but the sheer size introduces integration risk. The PR description is incomplete (all checkboxes unchecked, no testing confirmation), which is concerning for changes of this magnitude. While no critical bugs were found in the review, comprehensive platform-specific testing is essential before merge.
• Pay close attention to the new C++ tool calling implementation (sdk/runanywhere-commons/src/features/llm/tool_calling.cpp) for edge cases, the diffusion and VLM backend integrations, and verify that all platform-specific example apps have been tested on their respective platforms

Important Files Changed

Filename	Overview
sdk/runanywhere-commons/include/rac/features/llm/rac_tool_calling.h	Comprehensive tool calling API header with support for multiple LLM formats (DEFAULT, LFM2), includes parsing, prompt formatting, and JSON utilities
sdk/runanywhere-commons/src/features/llm/tool_calling.cpp	Single source of truth implementation for tool calling with manual JSON parsing, format detection, and comprehensive edge case handling
sdk/runanywhere-commons/include/rac/features/diffusion/rac_diffusion_types.h	Diffusion (image generation) type definitions with support for text-to-image, image-to-image, and inpainting workflows
sdk/runanywhere-commons/include/rac/features/vlm/rac_vlm_types.h	Vision-Language Model (VLM) type definitions for multimodal AI capabilities with image understanding
sdk/runanywhere-react-native/packages/core/cpp/third_party/nlohmann/json.hpp	Third-party nlohmann/json v3.11.3 library (24,765 lines) for JSON parsing in React Native bridge
examples/react-native/RunAnywhereAI/src/utils/mathParser.ts	Secure math expression evaluator using recursive descent parser (avoids eval), supports basic arithmetic with proper precedence
sdk/runanywhere-commons/src/backends/llamacpp/rac_vlm_llamacpp.cpp	LlamaCPP backend for VLM with image embedding, chat template handling, and multimodal prompt construction
sdk/runanywhere-commons/src/features/diffusion/diffusion_component.cpp	Diffusion component implementation with model lifecycle, backend routing, and generation pipeline orchestration
sdk/runanywhere-commons/src/features/vlm/vlm_component.cpp	VLM component with initialization, health checks, and backend integration for vision-language tasks

Sequence Diagram

sequenceDiagram
    participant App as Example App
    participant SDK as Platform SDK<br/>(Swift/Kotlin/Flutter/RN)
    participant Bridge as Platform Bridge
    participant Commons as runanywhere-commons<br/>(C++)
    participant LLM as LLM Backend

    Note over App,LLM: Tool Calling Flow

    App->>SDK: generateWithTools(prompt, tools, options)
    SDK->>Bridge: buildInitialPrompt()
    Bridge->>Commons: rac_tool_call_build_initial_prompt()
    Commons-->>Bridge: formatted prompt with tool instructions
    Bridge-->>SDK: system prompt + user prompt
    
    SDK->>LLM: generate(formatted_prompt)
    LLM-->>SDK: raw output with tool call tags
    
    SDK->>Bridge: parseToolCall(llm_output)
    Bridge->>Commons: rac_tool_call_parse()
    Note over Commons: Auto-detect format<br/>(DEFAULT vs LFM2)
    Note over Commons: Parse JSON with<br/>edge case handling
    Commons-->>Bridge: ToolCall{name, args, callId}
    Bridge-->>SDK: parsed tool call
    
    SDK->>App: toolRegistry.execute(toolName, args)
    App-->>SDK: tool result
    
    SDK->>Bridge: buildFollowupPrompt()
    Bridge->>Commons: rac_tool_call_build_followup_prompt()
    Commons-->>Bridge: followup prompt with result
    Bridge-->>SDK: formatted followup
    
    SDK->>LLM: generate(followup_prompt)
    LLM-->>SDK: final response
    SDK-->>App: GenerationResult{text, toolCalls, toolResults}

Loading

[coderabbitai]

Important

Review skipped

Too many files!

This PR contains 218 files, which is 68 over the limit of 150.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch dev

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[ellipsis-dev]

⚠️ This PR is too big for Ellipsis, but support for larger PRs is coming soon. If you want us to prioritize this feature, let us know at help@ellipsis.dev

---

Generated with ❤️ by ellipsis.dev

[shubhammalhotra28]

@coderabbitai please do a detailed review. Thanks!

[shubhammalhotra28]

@greptile please do a detailed review. Thanks!

[coderabbitai]

@shubhammalhotra28, I'll conduct a detailed review of PR #345 for you. Let me analyze all the changes thoroughly.

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[shubhammalhotra28]

Merging dev to main to add the features for :

• Tool Calling in all the 4 SDK's
• VLM support in ios
• Diffusion model support in ios using coreML