SentencePiece for Android not found

[lenyturmel]

Hi all,

I develop an Android application with Kotlin.
I am trying to use ai.djl to create a sentencepiece tokenizer from files.

I added the following dependencies to my project:

    implementation("ai.djl:api")  
    implementation("ai.djl.android:core")  
    implementation("ai.djl.sentencepiece:sentencepiece")  
    implementation("ai.djl.android:tokenizer-native:0.33.0")  
    implementation("ai.djl.android:pytorch-native:0.33.0")

I added libdjl_tokenizer.so to my JNI liv forlder (and the native C++_shared.so) and I loaded the libs using System.load.Library
I try to create the sentencepiece tokenizer as follow:

import ai.djl.sentencepiece.SpTokenizer
//...
 val modelInputStream: InputStream = context.assets.open("sentencepiece.bpe.model")
 tokenizer = SpTokenizer(modelInputStream)

It fails with the error:
java.io.IOException: Resource not found in classpath: native/lib/linux-aarch64/libsentencepiece_native.so

First I wonder why it is looking for a lib for the ptf linux-aarch64 ...
Then, I opened sentencepiece-0.33.0.jar to look for native C++ file for my project but there is no native lib for the android plateform. There is only the libs for Linux_aarch64, linux-x86_64, Osx-x86_64, and win-x86_64. There is no lib for arm64-v8a

Could you please help me ?
What I am doing wrong?
Could you please tell me how to create and run a SpTokenizer for the Android (arm64-v8a)?

Any help would be very appreciated ;)
Thanks

[farmaker47]

@lenyturmel where did you find the libdjl_tokenizer.so and the C++_shared.so ? I want to build or download them.

[lenyturmel]

I don't remember having any problems withlibdjl_tokenizer.so
My issue was about the native lib libsentencepiece_native.so

[farmaker47]

@lenyturmel apologies. Maybe I did not express correctly what I wanted.
I need the c++_shared.so file so I can replicate your issue. How did you build it or from where did you downloaded it?
Thanks

[lenyturmel]

okay, no problem
I didn't want to waste too much time because building a native lib can take quite a long time and I wanted to quickly do a feasibility test to run my models on Android. So I did not try to build the .so which is missing in DJL for Android
But, if it could help you, I found another lib that meets my needs:
https://github.com/shubham0204/Sentence-Embeddings-Android/blob/main/README.md
And there is some detailed explanations about building the native libs and package for Android with Rust here : https://proandroiddev.com/from-python-to-android-hf-sentence-transformers-embeddings-1ecea0ce94d8

Probably, I will try to use DSL later and to build my own native lib for Android, but not this time ;)

I am very interested in following your work if you succeed in building the lib and using DJL on Android. So, feel free to keep me in touch.
Thx

[farmaker47]

Thanks!
I will keep you posted.