[Swift-example][Playground] [Swift-SDK] Introducing YapRun#400
Merged
sanchitmonga22 merged 38 commits intomainfrom Feb 24, 2026
Merged
[Swift-example][Playground] [Swift-SDK] Introducing YapRun#400sanchitmonga22 merged 38 commits intomainfrom
sanchitmonga22 merged 38 commits intomainfrom
Conversation
- Enabled local binaries for development in Package.swift. - Added VoiceDictationManagementView and ViewModel for managing dictation settings and history. - Introduced FlowSessionManager to handle audio recording and transcription flow. - Created shared constants and data bridge for inter-process communication between the main app and keyboard extension. - Updated Info.plist for keyboard extension with necessary permissions and configurations. - Integrated deep linking to start dictation sessions from the keyboard. This commit lays the groundwork for a fully functional voice dictation feature in the RunAnywhereAI iOS app.
|
Important Review skippedToo many files! This PR contains 162 files, which is 12 over the limit of 150. ⛔ Files ignored due to path filters (30)
📒 Files selected for processing (162)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
sanchitmonga22
changed the title
- Removed hardcoded color schemes in multiple views, replacing them with adaptive colors from AppColors. - Updated ContentView, ModelCardView, NotepadView, and various onboarding steps to use AppColors for text and background styles. - Added a new AppTab enum for managing tab selection in the iOS app. - Introduced a new network client entitlement in the app's entitlements file. - Ensured that the app respects the system's light/dark mode settings across various views.
- Added WhisperKit as a new library for speech-to-text processing using Apple Neural Engine. - Introduced WhisperKitRuntime target and integrated it into the main application. - Updated ModelRegistry to include WhisperKit models and their respective handling. - Enhanced RunAnywhere SDK to support model registration and discovery for WhisperKit. - Updated various components to utilize WhisperKit for STT, ensuring compatibility with existing frameworks. - Refactored model management to accommodate WhisperKit's directory-based model structure.
- Added RunAnywhereWhisperKit SPM product for Apple Neural Engine STT - Changed WhisperKit dependency from local path to remote URL for SPM consumers - Fixed model download persistence: added RAC_FRAMEWORK_WHISPERKIT to discovery scan - Fixed registerModel() race condition with flushPendingRegistrations() - Emitted Swift STT events for WhisperKit model load/unload - Updated checksums for v0.19.2 xcframeworks (iOS + macOS) Co-authored-by: Cursor <cursoragent@cursor.com>
shubhammalhotra28
force-pushed
the
smonga/wispr_flow_clone
branch
from
2c9c857 to
87fcb95
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Playground/on-device-browser-agent/src/background/agents/navigator-agent.ts
Fixed
Show fixed
Hide fixed
Playground/on-device-browser-agent/src/background/agents/navigator-agent.ts
Fixed
Show fixed
Hide fixed
Playground/on-device-browser-agent/src/background/agents/state-machines/youtube.ts
Fixed
Show fixed
Hide fixed
Playground/on-device-browser-agent/src/background/agents/state-machines/youtube.ts
Fixed
Show fixed
Hide fixed
shubhammalhotra28
force-pushed
the
main
branch
from
3f2c8d2 to
eaf9e08
Compare
- Fixed framework_to_string() in telemetry to serialize whisperkit_coreml, coreml, mlx - Fixed STT component to configure actual_framework before model load - Pass framework from ModelInfo through CppBridge.STT.loadModel() - Updated checksums for v0.19.3 xcframeworks (iOS + macOS) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Added CFBundleVersion to build-ios.sh Info.plist templates (RACommons, backends) - Added CFBundleVersion + CFBundleShortVersionString to onnxruntime macOS plist in create-onnxruntime-xcframework.sh - Added MinimumOSVersion to pre-built onnxruntime iOS Info.plists - Updated Package.swift version to 0.19.4 with new checksums - Updated sdk/runanywhere-swift/VERSION to 0.19.4 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rameworks - Restructure xcframeworks from .framework format to plain .a library format to prevent SPM/Xcode from embedding static libraries as dynamic frameworks - Split onnxruntime into iOS (static) and macOS (dynamic) xcframeworks since xcframeworks can't mix library types - Make all library products explicitly .static in Package.swift - Add build phase to strip SPM-generated framework stubs from app bundle - Update build-ios.sh and create-onnxruntime-xcframework.sh to produce library-format xcframeworks going forward This fixes: "Invalid Bundle - does not support minimum OS Version" and "Upload Symbols Failed" errors during App Store validation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
[Swift-SDK] [Swift-Example] Whisper kit integration
… errors type: .static on SPM library products prevents binary target linker flags from being forwarded to consuming apps, causing "Undefined symbol: _rac_*" errors when archiving. Reverting to automatic library type resolves this. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Library-format xcframeworks (.a files) cause linker errors when consumed by Xcode apps via remote SPM. Reverted release zips to framework-format (.framework bundles) which properly propagate linker flags. Local dev still uses library-format via useLocalBinaries=true. The YapRun strip build phase handles App Store embedding. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Smonga/whisper kit integration
Resolved 7 merge conflicts: - Package.swift: Keep WhisperKit product + ragProducts() + split ONNX binaries - RunAnywhereAIApp.swift: Keep both WhisperKit + RAG embedding models - CMakeLists.txt: Add RAG backend build + combined backend summary - build-ios.sh: Keep library format xcframeworks + macOS slices - tests/CMakeLists.txt: Combine integration tests + RAG GoogleTests - AudioCaptureManager.swift: Keep Bluetooth workaround + ceil() fix - RunAnywhere+ModelManagement.swift: Keep framework param for STT loading Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sanchitmonga22
changed the title
Contributor
Author
|
@greptile please review |
Contributor
Author
|
@coderabbitai please review |
|
Generated with ❤️ by ellipsis.dev |
|
✅ Actions performedReview triggered.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| logger.info("Received kill deep link — killing session and terminating") | ||
| Task { | ||
| await flowSession.killSession() | ||
| exit(0) |
Contributor
There was a problem hiding this comment.
exit(0) bypasses normal cleanup and can leave resources in inconsistent state. Consider using a graceful shutdown pattern instead.
Suggested change
| exit(0) | |
| await flowSession.killSession() | |
| // Allow the system to handle app termination gracefully |
Prompt To Fix With AI
This is a comment left during a code review.
Path: Playground/YapRun/YapRun/YapRunApp.swift
Line: 112
Comment:
`exit(0)` bypasses normal cleanup and can leave resources in inconsistent state. Consider using a graceful shutdown pattern instead.
```suggestion
await flowSession.killSession()
// Allow the system to handle app termination gracefully
```
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+144
to
+158
| let semaphore = DispatchSemaphore(value: 0) | ||
|
|
||
| Task { | ||
| do { | ||
| try await WhisperKitSTTService.shared.loadModel(modelId: id, modelFolder: path) | ||
| handle = Unmanaged.passRetained(WhisperKitSTTService.shared).toOpaque() | ||
| } catch { | ||
| let logger = SDKLogger(category: "WhisperKitCoreML") | ||
| logger.error("Failed to load WhisperKit CoreML model: \(error)") | ||
| handle = nil | ||
| } | ||
| semaphore.signal() | ||
| } | ||
|
|
||
| semaphore.wait() |
Contributor
There was a problem hiding this comment.
Using DispatchSemaphore.wait() blocks the calling thread until the async task completes. This can cause thread pool exhaustion if called from a cooperative thread (like Swift concurrency runtime threads) and violates Swift 6 strict concurrency rules. Consider restructuring to use async/await throughout the call chain.
Prompt To Fix With AI
This is a comment left during a code review.
Path: sdk/runanywhere-swift/Sources/WhisperKitRuntime/WhisperKitSTT.swift
Line: 144-158
Comment:
Using `DispatchSemaphore.wait()` blocks the calling thread until the async task completes. This can cause thread pool exhaustion if called from a cooperative thread (like Swift concurrency runtime threads) and violates Swift 6 strict concurrency rules. Consider restructuring to use async/await throughout the call chain.
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+184
to
+206
| let semaphore = DispatchSemaphore(value: 0) | ||
|
|
||
| Task { | ||
| do { | ||
| let service = Unmanaged<WhisperKitSTTService>.fromOpaque(handle).takeUnretainedValue() | ||
| let output = try await service.transcribe(data, options: sttOptions) | ||
|
|
||
| outResult.pointee.text = output.text.isEmpty ? nil : strdup(output.text) | ||
| outResult.pointee.confidence = output.confidence | ||
| outResult.pointee.processing_time_ms = Int64(output.metadata.processingTime * 1000) | ||
|
|
||
| if let lang = output.detectedLanguage { | ||
| outResult.pointee.detected_language = strdup(lang) | ||
| } | ||
|
|
||
| result = RAC_SUCCESS | ||
| } catch { | ||
| let logger = SDKLogger(category: "WhisperKitCoreML") | ||
| logger.error("WhisperKit CoreML transcribe failed: \(error)") | ||
| result = RAC_ERROR_INTERNAL | ||
| } | ||
| semaphore.signal() | ||
| } |
Contributor
There was a problem hiding this comment.
Same DispatchSemaphore blocking pattern here. Since this is called from C++ callbacks that need synchronous responses, the blocking is unavoidable, but be aware this violates Swift 6 concurrency guarantees and can cause deadlocks if called on @MainActor contexts.
Prompt To Fix With AI
This is a comment left during a code review.
Path: sdk/runanywhere-swift/Sources/WhisperKitRuntime/WhisperKitSTT.swift
Line: 184-206
Comment:
Same `DispatchSemaphore` blocking pattern here. Since this is called from C++ callbacks that need synchronous responses, the blocking is unavoidable, but be aware this violates Swift 6 concurrency guarantees and can cause deadlocks if called on `@MainActor` contexts.
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+162
to
+167
| MainActor.assumeIsolated { | ||
| guard let self else { return } | ||
| SharedDataBridge.shared.audioLevel = self.audioCapture.audioLevel | ||
| guard case .listening = self.sessionPhase else { return } | ||
| self.audioBuffer.append(data) | ||
| } |
Contributor
There was a problem hiding this comment.
MainActor.assumeIsolated is unsafe if the callback isn't actually on the main thread. AVAudioEngine's tap callback runs on a real-time audio thread. Check if audioCapture.startRecording's callback is guaranteed to dispatch to main queue, otherwise this will crash.
Prompt To Fix With AI
This is a comment left during a code review.
Path: Playground/YapRun/YapRun/Features/VoiceKeyboard/FlowSessionManager.swift
Line: 162-167
Comment:
`MainActor.assumeIsolated` is unsafe if the callback isn't actually on the main thread. AVAudioEngine's tap callback runs on a real-time audio thread. Check if `audioCapture.startRecording`'s callback is guaranteed to dispatch to main queue, otherwise this will crash.
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+169
to
+174
| MainActor.assumeIsolated { | ||
| guard let self else { return } | ||
| self.audioBuffer.append(data) | ||
| self.audioLevel = self.audioCapture.audioLevel | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
Same MainActor.assumeIsolated concern - verify the audio callback actually runs on main queue
Prompt To Fix With AI
This is a comment left during a code review.
Path: Playground/YapRun/YapRun/macOS/Services/MacDictationService.swift
Line: 169-174
Comment:
Same `MainActor.assumeIsolated` concern - verify the audio callback actually runs on main queue
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+180
to
+185
| MainActor.assumeIsolated { | ||
| guard let self else { return } | ||
| // Always update audio level for waveform display | ||
| SharedDataBridge.shared.audioLevel = self.audioCapture.audioLevel | ||
| // Only accumulate audio data during the explicit listening window | ||
| guard case .listening = self.sessionPhase else { return } |
Contributor
There was a problem hiding this comment.
Same MainActor.assumeIsolated concern - verify the audio callback actually runs on main queue
Prompt To Fix With AI
This is a comment left during a code review.
Path: examples/ios/RunAnywhereAI/RunAnywhereAI/Features/VoiceKeyboard/FlowSessionManager.swift
Line: 180-185
Comment:
Same `MainActor.assumeIsolated` concern - verify the audio callback actually runs on main queue
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+74
to
+77
| MainActor.assumeIsolated { | ||
| guard let self else { return } | ||
| self.audioBuffer.append(data) | ||
| self.audioLevel = self.audioCapture.audioLevel |
Contributor
There was a problem hiding this comment.
Same MainActor.assumeIsolated concern - verify the audio callback actually runs on main queue
Prompt To Fix With AI
This is a comment left during a code review.
Path: Playground/YapRun/YapRun/Features/Playground/PlaygroundViewModel.swift
Line: 74-77
Comment:
Same `MainActor.assumeIsolated` concern - verify the audio callback actually runs on main queue
How can I resolve this? If you propose a fix, please make it concise.macOS frameworks require Versions/A/ directory structure with symlinks. The flat iOS-style structure caused linker errors on macOS builds. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Framework-format xcframeworks for proper SPM linker flag propagation - Versioned macOS framework structure (Versions/A/ with symlinks) - Flat iOS framework structure (standard) - Fixes undefined symbol errors on both iOS and macOS remote SPM builds Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace stale "ONNXRuntimeBinary" dependency (which doesn't exist) with platform-conditional ONNXRuntimeiOSBinary/ONNXRuntimemacOSBinary targets. This was preventing SPM from resolving the package graph. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The LoRA functions (rac_llm_component_*_lora) are declared in headers but not yet implemented in the RACommons C++ library. The RAG functions (rac_rag_*) are implemented in RABackendRAG which is excluded from remote releases. Both cause undefined symbol linker errors. Add weak stub implementations in shim.c that return RAC_ERROR_NOT_IMPLEMENTED. When the real backends are available, the strong definitions will override these stubs automatically. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add weak stubs for rac_voice_agent_* and rac_whisperkit_coreml_stt_set_callbacks (not yet in RACommons binary) - Add ARCHS=arm64 to YapRun (xcframeworks are arm64-only) - Fix OTHER_LDFLAGS to use $(inherited) + -all_load array format (required to force-link all symbols from SPM binary targets) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
Author
|
@greptile please review |
Contributor
Author
|
@coderabbitai please review |
|
🧠 Learnings used✅ Actions performedReview triggered.
|
Contributor
Additional Comments (1)
Sync this struct definition with the source of truth. Prompt To Fix With AIThis is a comment left during a code review.
Path: sdk/runanywhere-swift/Sources/RunAnywhere/CRACommons/include/rac_model_types.h
Line: 82-91
Comment:
Struct field mismatch with C++ source. The C++ header at `sdk/runanywhere-commons/include/rac/infrastructure/model_management/rac_model_types.h` defines these fields as `relative_path` and `destination_path`, but this copy has `url` and `filename`. This inconsistency will cause memory corruption when C++ code writes to these structs and Swift code reads them.
Sync this struct definition with the source of truth.
How can I resolve this? If you propose a fix, please make it concise. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Brief description of the changes made.
Type of Change
Testing
Platform-Specific Testing (check all that apply)
Swift SDK / iOS Sample:
Kotlin SDK / Android Sample:
Flutter SDK / Flutter Sample:
React Native SDK / React Native Sample:
Playground:
Web SDK / Web Sample:
Labels
Please add the appropriate label(s):
SDKs:
Swift SDK- Changes to Swift SDK (sdk/runanywhere-swift)Kotlin SDK- Changes to Kotlin SDK (sdk/runanywhere-kotlin)Flutter SDK- Changes to Flutter SDK (sdk/runanywhere-flutter)React Native SDK- Changes to React Native SDK (sdk/runanywhere-react-native)Web SDK- Changes to Web SDK (sdk/runanywhere-web)Commons- Changes to shared native code (sdk/runanywhere-commons)Sample Apps:
iOS Sample- Changes to iOS example app (examples/ios)Android Sample- Changes to Android example app (examples/android)Flutter Sample- Changes to Flutter example app (examples/flutter)React Native Sample- Changes to React Native example app (examples/react-native)Web Sample- Changes to Web example app (examples/web)Checklist
Screenshots
Attach relevant UI screenshots for changes (if applicable):
Greptile Summary
This PR introduces YapRun - a new on-device voice dictation playground app built on the RunAnywhere SDK. It demonstrates production-ready voice-to-text capabilities across iOS and macOS platforms with a Wispr Flow-inspired architecture.
Key Changes
New Playground Application
Playground/YapRun/) with 60+ new Swift filesWhisperKit CoreML Integration
WhisperKitRuntimemodule for Apple Neural Engine STT accelerationrac_stt_whisperkit_coreml.h)Architecture Highlights
Issues Found
Critical
rac_model_types.hbetween Swift SDK copy and C++ source -url/filenamevsrelative_path/destination_pathwill cause memory corruptionKnown (Already Flagged)
MainActor.assumeIsolatedusage in audio callbacks (needs verification that callbacks are main-dispatched)DispatchSemaphore.wait()blocking in WhisperKit callbacks (Swift 6 violation but unavoidable for C interop)exit(0)usage on deep link (bypasses cleanup)Confidence Score: 3/5
rac_model_types.his a critical memory safety issue that must be fixed before merge. The other flagged issues (semaphores, MainActor.assumeIsolated, exit) were already discussed in previous reviews and represent known trade-offs or require verification rather than immediate fixes.sdk/runanywhere-swift/Sources/RunAnywhere/CRACommons/include/rac_model_types.hrequires immediate sync with C++ source to fix struct definition mismatchImportant Files Changed
Last reviewed commit: 89a0a6f