Skip to content

Commit f035f20

Browse files
shubham0204shubham0204
committed
core: add model2vec and sample app, restructure project
1 parent b2cd140 commit f035f20

File tree

75 files changed

+62834
-61528
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

75 files changed

+62834
-61528
lines changed

‎.gitattributes‎

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎.github/FUNDING.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
github: shubham0204
2+
custom: ['https://www.paypal.me/ShubhamPanchal0204/']

‎.github/workflows/build.yml‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
name: Build Artifacts
2+
3+
on:
4+
push:
5+
tags:
6+
- 'v*'
7+
8+
# For packages and software in the ubuntu-latest runner image
9+
# see https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md
10+
# For us, it already contains the JDK, Cargo and Python installations
11+
12+
jobs:
13+
build:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v3
17+
18+
- name: Setup Android NDK
19+
uses: nttld/setup-ndk@v1
20+
id: setup-ndk
21+
with:
22+
ndk-version: r27c
23+
add-to-path: false
24+
env:
25+
ANDROID_NDK_HOME: ${{ steps.setup-ndk.outputs.ndk-path }}
26+
27+
- name: Install Rust targets for Android
28+
run: rustup target add aarch64-linux-android armv7-linux-androideabi i686-linux-android x86_64-linux-android
29+
30+
- name: Build Rust code
31+
run:
32+
./gradlew cargoBuild --stacktrace
33+
34+
- name: Build AAR for sentence_embeddings module
35+
run: ./gradlew :sentence_embeddings:assembleRelease --stacktrace
36+
37+
- name: Build AAR for model2vec module
38+
run: ./gradlew :model2vec:assembleRelease --stacktrace
39+
40+
- name: Decode keystore
41+
env:
42+
ENCODED_STRING: ${{ secrets.KEYSTORE_BASE_64 }}
43+
RELEASE_KEYSTORE_PASSWORD: ${{ secrets.RELEASE_KEYSTORE_PASSWORD }}
44+
RELEASE_KEYSTORE_ALIAS: ${{ secrets.RELEASE_KEYSTORE_ALIAS }}
45+
RELEASE_KEY_PASSWORD: ${{ secrets.RELEASE_KEY_PASSWORD }}
46+
run: |
47+
echo $ENCODED_STRING > keystore-b64.txt
48+
base64 -d keystore-b64.txt > keystore.jks
49+
50+
- name: Build APK for app module
51+
env:
52+
RELEASE_KEYSTORE_PASSWORD: ${{ secrets.RELEASE_KEYSTORE_PASSWORD }}
53+
RELEASE_KEYSTORE_ALIAS: ${{ secrets.RELEASE_KEYSTORE_ALIAS }}
54+
RELEASE_KEY_PASSWORD: ${{ secrets.RELEASE_KEY_PASSWORD }}
55+
run: ./gradlew :app:assembleRelease --stacktrace
56+
57+
- name: Build APK for app-model2vec module
58+
env:
59+
RELEASE_KEYSTORE_PASSWORD: ${{ secrets.RELEASE_KEYSTORE_PASSWORD }}
60+
RELEASE_KEYSTORE_ALIAS: ${{ secrets.RELEASE_KEYSTORE_ALIAS }}
61+
RELEASE_KEY_PASSWORD: ${{ secrets.RELEASE_KEY_PASSWORD }}
62+
run: ./gradlew :app-model2vec:assembleRelease --stacktrace
63+
64+
- name: Create a release
65+
uses: actions/create-release@v1
66+
id: create_release
67+
with:
68+
tag_name: ${{ github.ref }}
69+
release_name: ${{ github.ref }}
70+
draft: false
71+
prerelease: false
72+
body_path: CHANGELOG.md
73+
env:
74+
GITHUB_TOKEN: ${{ github.token }}
75+
76+
- name: Upload AAR for sentence_embeddings module to release
77+
uses: actions/upload-release-asset@v1
78+
env:
79+
GITHUB_TOKEN: ${{ github.token }}
80+
with:
81+
upload_url: ${{ steps.create_release.outputs.upload_url }}
82+
asset_path: sentence_embeddings/build/outputs/aar/sentence_embeddings-release.aar
83+
asset_name: sentence_embeddings.aar
84+
asset_content_type: application/octet-stream
85+
86+
- name: Upload AAR for model2vec module to release
87+
uses: actions/upload-release-asset@v1
88+
env:
89+
GITHUB_TOKEN: ${{ github.token }}
90+
with:
91+
upload_url: ${{ steps.create_release.outputs.upload_url }}
92+
asset_path: model2vec/build/outputs/aar/model2vec-release.aar
93+
asset_name: model2vec.aar
94+
asset_content_type: application/octet-stream
95+
96+
- name: Upload APK for app module to release
97+
uses: actions/upload-release-asset@v1
98+
env:
99+
GITHUB_TOKEN: ${{ github.token }}
100+
with:
101+
upload_url: ${{ steps.create_release.outputs.upload_url }}
102+
asset_path: app/build/outputs/apk/release/app-release.apk
103+
asset_name: app_${{ github.ref_name }}.apk
104+
asset_content_type: application/vnd.android.package-archive
105+
106+
- name: Upload APK for app-model2vec module to release
107+
uses: actions/upload-release-asset@v1
108+
env:
109+
GITHUB_TOKEN: ${{ github.token }}
110+
with:
111+
upload_url: ${{ steps.create_release.outputs.upload_url }}
112+
asset_path: app-model2vec/build/outputs/apk/release/app-model2vec-release.apk
113+
asset_name: app_model2vec_${{ github.ref_name }}.apk
114+
asset_content_type: application/vnd.android.package-archive

‎CHANGELOG.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
- Move the Rust source code from the `libs` branch to the `main` branch: We now use the [rust-android-plugin](https://github.com/mozilla/rust-android-gradle/issues/29) to initiate `cargo build` from Gradle
2+
3+
- Removed Git LFS: The ONNX models present in `app/src/main/assets` have been removed from the repository. Instead, `app/build.gradle.kts` downloads the models and tokenizer configs from HuggingFace using `download_model.sh` shell script.
4+
5+
- Add [Model2Vec](https://huggingface.co/blog/Pringled/model2vec): Model2Vec provides static sentence-embeddings through a fast-lookup
6+
7+
- Remove Jitpack: A GitHub CI script now builds AARs for `model2vec` and `sentence_embeddings` Gradle modules that can be included in other projects

‎README.md‎

Lines changed: 17 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -10,54 +10,38 @@
1010

1111
## Updates
1212

13-
- 2024-08: Along with `token_ids` and `attention_mask`, the native library now also returns `token_type_ids` to support additional models like the `bge-small-en-v1.5` (issue #3)
13+
### 2025-03
1414

15-
## Supported Models
15+
- Move the Rust source code from the `libs` branch to the `main` branch: We now use the [rust-android-plugin](https://github.com/mozilla/rust-android-gradle/issues/29) to initiate `cargo build` from Gradle
1616

17-
- [`all-minilm-l6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/tree/main)
18-
- [`bge-small-en-v1.5`](https://huggingface.co/BAAI/bge-small-en-v1.5)
19-
- [`snowflake-arctic-embed-s`](https://huggingface.co/Snowflake/snowflake-arctic-embed-s)
17+
- Removed Git LFS: The ONNX models present in `app/src/main/assets` have been removed from the repository. Instead, `app/build.gradle.kts` downloads the models and tokenizer configs from HuggingFace using `download_model.sh` shell script.
2018

21-
To add more models, refer the [Adding New Models](#adding-new-models) section.
19+
- Add [Model2Vec](https://huggingface.co/blog/Pringled/model2vec): Model2Vec provides static sentence-embeddings through a fast-lookup
2220

23-
## Setup
21+
- Remove Jitpack: A GitHub CI script now builds AARs for `model2vec` and `sentence_embeddings` Gradle modules that can be included in other projects
2422

25-
### 1. Add the Jitpack repository to `settings.gradle.kts`
23+
### 2024-08
2624

27-
The library is hosted with [Jitpack](https://jitpack.io/). Add the `jitpack.io` repository in `settings.gradle.kts` for Gradle to search Jitpack packages,
25+
- Along with `token_ids` and `attention_mask`, the native library now also returns `token_type_ids` to support additional models like the `bge-small-en-v1.5` (issue #3)
2826

29-
```kotlin
30-
dependencyResolutionManagement {
31-
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
32-
repositories {
33-
google()
34-
mavenCentral()
35-
maven{ url = uri("https://jitpack.io") }
36-
}
37-
}
38-
```
27+
## Supported Models
3928

40-
or with Groovy build scripts,
29+
- [`all-minilm-l6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/tree/main)
30+
- [`bge-small-en-v1.5`](https://huggingface.co/BAAI/bge-small-en-v1.5)
31+
- [`snowflake-arctic-embed-s`](https://huggingface.co/Snowflake/snowflake-arctic-embed-s)
4132

42-
```groovy
43-
dependencyResolutionManagement {
44-
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
45-
repositories {
46-
google()
47-
mavenCentral()
48-
maven { url "https://jitpack.io" }
49-
}
50-
}
51-
```
33+
To add more models, refer the [Adding New Models](#adding-new-models) section.
5234

53-
### 2. Add the dependency to `build.gradle.kts`
35+
## Installation
5436

55-
Add the `Sentence-Embeddings-Android` dependency to `build.gradle.kts`,
37+
The AARs for the `sentence_embeddings` and `model2vec` modules are available in the [Releases](https://github.com/shubham0204/Sentence-Embeddings-Android/releases) which can be downloaded. Add the AARs to the `app/libs` directory and then in `app/build.gradle.kts`,
5638

5739
```kotlin
5840
dependencies {
5941
// ...
60-
implementation("com.github.shubham0204:Sentence-Embeddings-Android:0.0.5")
42+
// Add one or both of them as needed
43+
implementation(file("libs/sentence_embeddings.aar"))
44+
implementation(file("libs/model2vec.aar"))
6145
// ...
6246
}
6347
```

‎app-model2vec/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/build

‎app-model2vec/build.gradle.kts‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
plugins {
2+
alias(libs.plugins.android.application)
3+
alias(libs.plugins.jetbrains.kotlin.android)
4+
alias(libs.plugins.kotlin.compose)
5+
}
6+
7+
android {
8+
namespace = "com.projects.shubham0204.demo_model2vec"
9+
compileSdk = 35
10+
11+
defaultConfig {
12+
applicationId = "com.projects.shubham0204.demo_model2vec"
13+
minSdk = 24
14+
targetSdk = 35
15+
versionCode = 1
16+
versionName = "1.0"
17+
18+
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
19+
}
20+
21+
signingConfigs {
22+
create("release") {
23+
storeFile = file("../keystore.jks")
24+
storePassword = System.getenv("RELEASE_KEYSTORE_PASSWORD")
25+
keyAlias = System.getenv("RELEASE_KEYSTORE_ALIAS")
26+
keyPassword = System.getenv("RELEASE_KEY_PASSWORD")
27+
}
28+
}
29+
30+
buildTypes {
31+
release {
32+
isMinifyEnabled = false
33+
proguardFiles(
34+
getDefaultProguardFile("proguard-android-optimize.txt"),
35+
"proguard-rules.pro",
36+
)
37+
signingConfig = signingConfigs.getByName("release")
38+
}
39+
}
40+
compileOptions {
41+
sourceCompatibility = JavaVersion.VERSION_17
42+
targetCompatibility = JavaVersion.VERSION_17
43+
}
44+
kotlinOptions {
45+
jvmTarget = "17"
46+
}
47+
buildFeatures {
48+
compose = true
49+
}
50+
}
51+
52+
dependencies {
53+
implementation(libs.androidx.core.ktx)
54+
implementation(libs.androidx.lifecycle.runtime.ktx)
55+
implementation(libs.androidx.activity.compose)
56+
implementation(platform(libs.androidx.compose.bom))
57+
implementation(libs.androidx.ui)
58+
implementation(libs.androidx.ui.graphics)
59+
implementation(libs.androidx.ui.tooling.preview)
60+
implementation(libs.androidx.material3)
61+
62+
implementation(project(":model2vec"))
63+
}

‎app-model2vec/proguard-rules.pro‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Add project specific ProGuard rules here.
2+
# You can control the set of applied configuration files using the
3+
# proguardFiles setting in build.gradle.
4+
#
5+
# For more details, see
6+
# http://developer.android.com/guide/developing/tools/proguard.html
7+
8+
# If your project uses WebView with JS, uncomment the following
9+
# and specify the fully qualified class name to the JavaScript interface
10+
# class:
11+
#-keepclassmembers class fqcn.of.javascript.interface.for.webview {
12+
# public *;
13+
#}
14+
15+
# Uncomment this to preserve the line number information for
16+
# debugging stack traces.
17+
#-keepattributes SourceFile,LineNumberTable
18+
19+
# If you keep the line number information, uncomment this to
20+
# hide the original source file name.
21+
#-renamesourcefileattribute SourceFile

‎app-model2vec/src/main/AndroidManifest.xml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
3+
4+
<application
5+
android:allowBackup="true"
6+
android:icon="@mipmap/ic_launcher"
7+
android:label="@string/app_name"
8+
android:roundIcon="@mipmap/ic_launcher_round"
9+
android:supportsRtl="true"
10+
android:theme="@style/Theme.SentenceEmbeddings">
11+
<activity
12+
android:name=".MainActivity"
13+
android:exported="true"
14+
android:label="@string/app_name"
15+
android:theme="@style/Theme.SentenceEmbeddings">
16+
<intent-filter>
17+
<action android:name="android.intent.action.MAIN" />
18+
19+
<category android:name="android.intent.category.LAUNCHER" />
20+
</intent-filter>
21+
</activity>
22+
</application>
23+
24+
</manifest>

‎app-model2vec/src/main/assets/embeddings.safetensors‎

28.8 MB
Binary file not shown.

0 commit comments

Comments
 (0)