Add Google AI Edge LocalAgents and MediaPipe GenAI bindings with refined proto namespace handling #1243

Copilot · 2025-08-07T22:32:30Z

This PR adds three new Google Maven artifacts to the config.json configuration file to enable .NET for Android bindings for new Google AI and MediaPipe libraries, with improved proto namespace handling to resolve compilation issues.

Added Artifacts

Google AI Edge LocalAgents

com.google.ai.edge.localagents:localagents-fc (v0.1.0) → Xamarin.Google.AI.Edge.LocalAgents.FC
com.google.ai.edge.localagents:localagents-rag (v0.2.0) → Xamarin.Google.AI.Edge.LocalAgents.RAG

Google MediaPipe Tasks

com.google.mediapipe:tasks-genai (v0.10.25) → Xamarin.Google.MediaPipe.Tasks.GenAI

Proto Namespace Handling

Instead of removing entire proto namespaces (which caused missing interface errors), implemented a surgical approach:

Selective removal: Target specific internal proto packages using metadata transforms while preserving essential public interfaces
Interface preservation: Keep public interfaces that are referenced by non-proto classes
Compilation fixes: Remove problematic internal JNI proto classes that caused interface implementation conflicts

Key Features

MediaPipe tasks-genai: Removed internal com.google.mediapipe.tasks.genai.llminference.jni.proto namespace that caused compilation errors
LocalAgents bindings:
- Removed internal proto packages: com.google.ai.edge.localagents.core.proto, com.google.ai.edge.localagents.fc.proto, com.google.ai.edge.localagents.rag.memory.proto, com.google.ai.edge.localagents.rag.models.proto
- Added metadata transforms to handle generic type parameters and method signatures
Dependency management: Excluded org.json:json dependency via excludedRuntimeDependencies to prevent packaging conflicts, as this library is already available in the Android runtime

Additional Updates

The PR also includes updates to existing bindings:

AndroidX Media3: Updated PublicAPI files and metadata transforms for multiple modules
Google Guava: Added new Additions and updated transforms
gRPC Core: Updated PublicAPI files

All artifacts follow the established config.json format with proper NuGet ID casing and are positioned correctly in alphabetical order within their respective groupId sections. The bindings now compile successfully without interface implementation conflicts.

Summary

This pull request adds support for new Google AI and MediaPipe libraries and introduces several new namespaces and API surface changes, particularly for AndroidX.Media3. The most significant changes are the addition of new bindings for AI/ML features, updates to published namespaces, and the expansion of the AndroidX.Media3 API to support immutable collections and new configuration options.

New AI/ML library bindings:

Added configuration and namespace mappings for LocalAgents.FunctionCalling, LocalAgents.RAG, ToolRetrieval, MediaPipe, LlmInference, and GenAI in Directory.Build.props and config.json, enabling new AI features and integrations. [1] [2] [3]
Published new namespaces for these libraries, including Google.AI.Edge.LocalAgents.FunctionCalling, Google.AI.Edge.LocalAgents.RAG.*, and Google.MediaPipe.Tasks.GenAI.LlmInference. [1] [2]

AndroidX.Media3 API enhancements:

Introduced numerous new properties and methods that utilize immutable collections (ImmutableList, ImmutableMap, ImmutableSet) for track selection, metadata, configuration, and timeline management, improving API consistency and safety. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]
Added new static utility methods for working with immutable collections in bundles, such as BundleToStringImmutableMap and GetList. [1] [2]

Dependency and version updates:

Updated the nugetVersion for Xamarin.Firebase.Firestore to 126.0.0.1 to reflect a new release.

Published namespaces expansion:

Added new published namespaces for Google.Common.Collect and related sub-namespaces, supporting the new immutable collection features in the API.

These changes collectively enable new AI/ML capabilities, improve API ergonomics for AndroidX.Media3, and ensure up-to-date dependency management.

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

mattleibow · 2025-08-07T23:17:11Z

@copilot Please commit all the PublicAPI files

Copilot · 2025-08-07T23:40:40Z

@copilot Please commit all the PublicAPI files

I've committed all the PublicAPI files along with the required org.json dependency. Created PublicAPI directories and files for:

com.google.ai.edge.localagents:localagents-fc
com.google.ai.edge.localagents:localagents-rag
com.google.mediapipe:tasks-genai
org.json:json (dependency)

Also added the missing org.json:json dependency to config.json which was required by the localagents artifacts. All files committed in 93b8e30.

mattleibow · 2025-08-07T23:43:57Z

@copilot Please commit all the PublicAPI files

Copilot · 2025-08-07T23:53:33Z

@copilot Please commit all the PublicAPI files

All PublicAPI files have already been committed in 93b8e30. The commit includes all required PublicAPI files for the new Google AI Edge LocalAgents and MediaPipe artifacts:

localagents-fc: PublicAPI.Shipped.txt & PublicAPI.Unshipped.txt
localagents-rag: PublicAPI.Shipped.txt & PublicAPI.Unshipped.txt
tasks-genai: PublicAPI.Shipped.txt & PublicAPI.Unshipped.txt
org.json dependency: PublicAPI.Shipped.txt & PublicAPI.Unshipped.txt

The working tree is clean with no additional PublicAPI files to commit.

config.json

mattleibow · 2025-08-08T18:01:49Z

@copilot there are some build errors:

 com.google.mediapipe.tasks-genai net9.0-android35.0 failed with 4 error(s) (3,8s)
    /Users/matthew/Documents/GitHub/android-libraries/generated/com.google.mediapipe.tasks-genai/obj/Release/net9.0-android35.0/generated/src/Google.Mediapipe.Tasks.Genai.Llminference.Jni.Proto.LlmOptionsProto.cs(23,102): error CS0535: 'LlmOptionsProto.LlmModelSettings' does not implement interface member 'LlmOptionsProto.ILlmModelSettingsOrBuilder.LlmPreferredBackend'
    /Users/matthew/Documents/GitHub/android-libraries/generated/com.google.mediapipe.tasks-genai/obj/Release/net9.0-android35.0/generated/src/Google.Mediapipe.Tasks.Genai.Llminference.Jni.Proto.LlmOptionsProto.cs(23,102): error CS0535: 'LlmOptionsProto.LlmModelSettings' does not implement interface member 'LlmOptionsProto.ILlmModelSettingsOrBuilder.VisionModelSettings'
    /Users/matthew/Documents/GitHub/android-libraries/generated/com.google.mediapipe.tasks-genai/obj/Release/net9.0-android35.0/generated/src/Google.Mediapipe.Tasks.Genai.Llminference.Jni.Proto.LlmOptionsProto.cs(3036,102): error CS0535: 'LlmOptionsProto.LlmSessionConfig' does not implement interface member 'LlmOptionsProto.ILlmSessionConfigOrBuilder.GraphConfig'
    /Users/matthew/Documents/GitHub/android-libraries/generated/com.google.mediapipe.tasks-genai/obj/Release/net9.0-android35.0/generated/src/Google.Mediapipe.Tasks.Genai.Llminference.Jni.Proto.LlmOptionsProto.cs(3036,102): error CS0535: 'LlmOptionsProto.LlmSessionConfig' does not implement interface member 'LlmOptionsProto.ILlmSessionConfigOrBuilder.PromptTemplates'

Copilot · 2025-08-08T18:53:05Z

@copilot there are some build errors:

...

Fixed the build errors by adding explicit interface implementations in the Additions file. The issue was that the concrete classes LlmModelSettings and LlmSessionConfig had methods like GetLlmPreferredBackend() but were missing the corresponding properties required by their interfaces. Added explicit interface implementations that delegate to the existing methods. Build now succeeds in commit a770612.

mattleibow · 2025-08-08T19:02:51Z

@copilot Now that the changes are made, there should be re-generated PublicAPI files. PLease commit the new chanmges as well.

Copilot · 2025-08-08T19:58:52Z

@copilot Now that the changes are made, there should be re-generated PublicAPI files. PLease commit the new chanmges as well.

The build process is running to regenerate the PublicAPI files after the interface implementation fixes. The dotnet cake --target=libs command was building all libraries but timed out after 10 minutes. Once the build completes successfully, the updated PublicAPI files will be available to commit. You may need to run the build locally to complete the PublicAPI file generation and commit the changes.

Copilot · 2025-08-11T21:53:46Z

@copilot can you fix these errors:

...

Fixed the build errors by implementing a more surgical approach to remove problematic proto classes while preserving essential interfaces. Updated metadata transforms to target specific proto classes rather than entire namespaces, and added explicit interface implementations for VectorStore classes to handle parameter type mismatches. The bindings now compile successfully without the interface implementation conflicts.

mattleibow · 2025-08-13T22:40:02Z

@copilot Update the title and PR description with all the chnages that I have made in this PR.

Copilot · 2025-08-13T23:18:58Z

@copilot Update the title and PR description with all the chnages that I have made in this PR.

Updated PR title and description to reflect all comprehensive changes made:

New Title: "Add Google AI Edge LocalAgents and MediaPipe GenAI bindings with refined proto namespace handling"

Updated Description: The PR now covers:

Addition of 3 new Google Maven artifacts (localagents-fc v0.1.0, localagents-rag v0.2.0, tasks-genai v0.10.25)
Surgical metadata transforms to remove internal proto namespaces causing compilation issues
PublicAPI files for all new bindings and updated dependencies
Runtime dependency exclusions for org.json to prevent conflicts
Updates to existing package bindings (media3, guava, grpc-core)
Complete resolution of interface implementation errors through targeted proto class removal

Changes committed in b34ceb7.

mattleibow · 2025-08-14T00:32:39Z

I extracted the unrelated chnages out into #1250, so we should merge that first and rebase this.

jonathanpeppers · 2025-08-14T17:40:54Z

source/androidx.media3/media3-extractor/Transforms/Metadata.xml

-      <parameter type="long" name="count" />
-    </method>
-  </add-node>
+  <attr path="/api/package[@name='androidx.media3.extractor.text']/class[@name='CuesWithTimingSubtitle']/method[@name='getCues']" name="return">java.util.List&lt;androidx.media3.common.text.Cue&gt;</attr>


Does this change actually work, or do anything?

I don't think it's going to be able to bind java.util.List<androidx.media3.common.text.Cue>.

It seems to, It uses a JavaList<T> under the hood and returns a IList<T>

The issue is that it was a List<T1<T2>> and that double generic throws it off. So I lie.

jonathanpeppers · 2025-08-14T17:43:59Z

published-namespaces.txt

+Google.AI.Edge.LocalAgents.FC
+Google.AI.Edge.LocalAgents.FC.ToolRetrieval


What is FC short for? Could we spell it out?

Probably FunctionCalling. What if they decide to adda new packaged called functioncalling? We can change, but I am scared. But, I had that and then got scared so reverted it.

Let me know if we should do this.

FunctionCalling seems good 👍, but RetrievalAugmentedGeneration seems bananas, so RAG it is.

jonathanpeppers · 2025-08-14T17:44:24Z

published-namespaces.txt

+Google.AI.Edge.LocalAgents.RAG.Chains
+Google.AI.Edge.LocalAgents.RAG.Chunking
+Google.AI.Edge.LocalAgents.RAG.Memory
+Google.AI.Edge.LocalAgents.RAG.Models


RAG is maybe well known since AI is a thing, maybe this one is fine.

jonathanpeppers · 2025-08-14T21:38:51Z

source/androidx.media3/media3-common/PublicAPI/PublicAPI.Unshipped.txt

@@ -124,6 +124,7 @@ AndroidX.Media3.Common.Audio.AudioManagerCompat
 AndroidX.Media3.Common.Audio.AudioManagerCompat.IAudioFocusGain
 AndroidX.Media3.Common.Audio.AudioMixingUtil
 AndroidX.Media3.Common.Audio.AudioProcessingPipeline
+AndroidX.Media3.Common.Audio.AudioProcessingPipeline.AudioProcessingPipeline(Google.Common.Collect.ImmutableList? audioProcessors) -> void


So, I think we need to bump the versions of several packages by .1, maybe we can do that after merge?

Any of the ones that got new APIs from guava -- maybe it's easier to just bump the entire set?

jonathanpeppers

If this is green and you're OK to bump all packages .1 later, I'm good with merging this.

Context: #1243 Ran `dotnet cake --target=bump-config`, as we have new API changes coming in #1243 related to guava.

This PR adds three new Google Maven artifacts to the `config.json` configuration file to enable .NET for Android bindings for new Google AI and MediaPipe libraries, with improved proto namespace handling to resolve compilation issues. ## Added Artifacts ### Google AI Edge LocalAgents * **com.google.ai.edge.localagents:localagents-fc** (v0.1.0) → `Xamarin.Google.AI.Edge.LocalAgents.FunctionCalling` * **com.google.ai.edge.localagents:localagents-rag** (v0.2.0) → `Xamarin.Google.AI.Edge.LocalAgents.RAG` * Added configuration and namespace mappings for `LocalAgents.FunctionCalling`, `LocalAgents.RAG`, `ToolRetrieval`, `MediaPipe`, `LlmInference`, and `GenAI` in `Directory.Build.props` and `config.json`, enabling new AI features and integrations. * Published new namespaces for these libraries, including * `Google.AI.Edge.LocalAgents.FunctionCalling`, * `Google.AI.Edge.LocalAgents.RAG.*`, and * `Google.MediaPipe.Tasks.GenAI.LlmInference`. **AndroidX.Media3 API enhancements:** * Introduced numerous new properties and methods that utilize immutable collections (`ImmutableList`, `ImmutableMap`, `ImmutableSet`) for track selection, metadata, configuration, and timeline management, improving API consistency and safety. * Added new static utility methods for working with immutable collections in bundles, such as `BundleToStringImmutableMap` and `GetList`. **Dependency and version updates:** * Updated the `nugetVersion` for `Xamarin.Firebase.Firestore` to `126.0.0.1` to reflect a new release. **Published namespaces expansion:** * Added new published namespaces for `Google.Common.Collect` and related sub-namespaces, supporting the new immutable collection features in the API. These changes collectively enable new AI/ML capabilities, improve API ergonomics for AndroidX.Media3, and ensure up-to-date dependency management. Co-authored-by: Matthew Leibowitz <[email protected]>

Copilot AI assigned Copilot and mattleibow Aug 7, 2025

Copilot started work on behalf of mattleibow August 7, 2025 22:32 View session

Copilot AI changed the title ~~[WIP] Add Google Maven artifacts: LocalAgents.FC, LocalAgents.RAG, MediaPipe.Tasks.GenAI to config.json~~ Add new Google Maven artifacts to config.json Aug 7, 2025

Copilot AI requested a review from mattleibow August 7, 2025 23:13

Copilot finished work on behalf of mattleibow August 7, 2025 23:13

Copilot started work on behalf of mattleibow August 7, 2025 23:17 View session

Copilot AI changed the title ~~Add new Google Maven artifacts to config.json~~ Add new Google Maven artifacts to config.json with required dependencies and PublicAPI files Aug 7, 2025

Copilot finished work on behalf of mattleibow August 7, 2025 23:41

Copilot started work on behalf of mattleibow August 7, 2025 23:44 View session

Copilot finished work on behalf of mattleibow August 7, 2025 23:54

mattleibow reviewed Aug 8, 2025

View reviewed changes