ai.djl.engine.EngineException: Failed to load PyTorch native library

[midgard-as]

Hello,

I have been trying to use DJL in my java project but i could not load PyTorch native library.

You can see my POM.xml :attached.
POM.txt

I have cuda 11.1 installed and I'm on Ubuntu 18.4. I also tried with pyorch-native-auto engine but it gave the same error !

Do you have any idea what the problem could be ?

Thank yu very much !

[frankfliu]

@midgard-as

The pom.xml file looks fine to me, but I recommend you to bom to manage DJL versions.

What error message did you get? Can you shared stacktrace?
Can you clone djl repository and run the following command and see if work:

git clone https://github.com/deepjavalibrary/djl.git
cd djl
PYTORCH_VERSION=1.9.1 ./gradlew debugEnv -Dai.djl.default_engine=PyTorch

[LiorXu]

pytorch need be build in global enviorment? if it was in Anconda ,the project will be fail? Shoud l rebuild pytorch in the global enviroment?
This is my question:
[Exception in thread "main" ai.djl.engine.EngineException: Failed to load PyTorch native library
at ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:81)
at ai.djl.pytorch.engine.PtEngineProvider.getEngine(PtEngineProvider.java:40)
at ai.djl.engine.Engine.getEngine(Engine.java:177)
at ai.djl.Model.newInstance(Model.java:98)
at ai.djl.repository.zoo.BaseModelLoader.createModel(BaseModelLoader.java:187)
at ai.djl.repository.zoo.BaseModelLoader.loadModel(BaseModelLoader.java:149)
at ai.djl.repository.zoo.Criteria.loadModel(Criteria.java:166)
at ai.djl.repository.zoo.ModelZoo.loadModel(ModelZoo.java:132)
at com.example.djl.DjlApplication.main(DjlApplication.java:64)
Caused by: java.lang.IllegalStateException: No PyTorch native library matches your operating system: cpu-win-x86:1.11.0
at ai.djl.pytorch.jni.LibUtils.downloadPyTorch(LibUtils.java:446)
at ai.djl.pytorch.jni.LibUtils.findNativeLibrary(LibUtils.java:283)]

[frankfliu]

DJL doesn't need Anaconda.

It looks like you are running a 32bit system, which we don't support. I don't think PyTorch officially support x86 either.

[LiorXu]

thanks,l try to change the x64 system later

[midgard-as]

Hello,
Thanks for your answer! I tried to run the command line and it works without any problem, the build was a success.

I'm trying to launch the face detection example of DJL as a plugin of a stream app running on antmedia.

` The code on my plugin is :

private FaceBlur(){

    System.out.println("creating face blur");
double confThresh = 0.85f;
    double nmsThresh = 0.45f;
    double[] variance = {0.1f, 0.2f};
    int topK = 5000;
    int[][] scales = {{10, 16, 24}, {32, 48}, {64, 96}, {128, 192, 256}};
    int[] steps = {8, 16, 32, 64};

    FaceDetectionTranslator translator = new FaceDetectionTranslator(confThresh, nmsThresh, variance, topK, scales, steps);
    //Path modelPath = Paths.get("~/models/ultranet.zip");.optModelPath(modelPath)

    this.criteria = Criteria.builder()
                .setTypes(Image.class, DetectedObjects.class)
                .optModelUrls("https://resources.djl.ai/test-models/pytorch/ultranet.zip")
                .optTranslator(translator)
                .optProgress(new ProgressBar())
                .optEngine("PyTorch") // Use PyTorch engine
                .build();

    try { 
        this.model = this.criteria.loadModel();
        System.out.println("MODEL BUILT");
        } 
    catch (Exception e){ 
        System.out.println("error on model load " + e);
        }
}`

The built of the plugin works fine but when i try to run it i have this error :

creating face blur Loading: 100% |████████████████████████████████████████| error on model load ai.djl.engine.EngineException: Failed to load PyTorch native library

The environment where the plugin runs is quite scpecific as i need to have every dependencies of the build locally. I've seen that the cmd line i ran for testing installed a lot of .so file on the system. Should i include them on my project ? How can i change the path to find them on my project ?

Thank you!

[frankfliu]

@midgard-as
If you can run the gradlew command successfully, that means your system environment is fine. The error most likely related to how you package the jar files. You might run into this issue: #940

For package the pytorch native library in your plugin, you can see: https://github.com/deepjavalibrary/djl/blob/master/engines/pytorch/pytorch-engine/README.md#linux

I can take a look your project if you can shared it on github.

[midgard-as]

Hello,

I have tried different to check the META-INF folder as said in the issue you linked :

~/pytorch-engine/META-INF/services$ cat ai.djl.engine.EngineProvider 
ai.djl.pytorch.engine.PtEngineProvider

I feel like this is expected and there is not much to change.

Here is the full stacktrace i face:

error on model load ai.djl.engine.EngineException: Failed to load PyTorch native libraryai.djl.engine.EngineException: Failed to load PyTorch native library
        at ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:77)
        at ai.djl.pytorch.engine.PtEngineProvider.getEngine(PtEngineProvider.java:40)
        at ai.djl.engine.Engine.getEngine(Engine.java:168)
        at ai.djl.Model.newInstance(Model.java:98)
        at ai.djl.repository.zoo.BaseModelLoader.createModel(BaseModelLoader.java:187)
        at ai.djl.repository.zoo.BaseModelLoader.loadModel(BaseModelLoader.java:149)
        at ai.djl.repository.zoo.Criteria.loadModel(Criteria.java:166)
        at io.antmedia.filter.FaceBlur.<init>(FaceBlur.java:116)
        at io.antmedia.filter.FaceBlur.getInstance(FaceBlur.java:129)
        at io.antmedia.filter.BlurAdaptor.getFrameFilter(BlurAdaptor.java:308)
        at io.antmedia.filter.BlurAdaptor.update(BlurAdaptor.java:166)
        at io.antmedia.filter.BlurAdaptor.onVideoFrame(BlurAdaptor.java:81)
        at io.antmedia.filter.BlurAdaptor.onVideoPacket(BlurAdaptor.java:283)
        at io.antmedia.plugin.PacketFeeder.writePacket(PacketFeeder.java:43)
        at io.antmedia.plugin.PacketFeeder.writeVideoBuffer(PacketFeeder.java:95)
        at io.antmedia.enterprise.adaptive.WebRTCVideoForwarder.writeVideoBuffer(WebRTCVideoForwarder.java:93)
        at io.antmedia.enterprise.adaptive.WebRTCVideoForwarder$VirtualVideoDecoder.lambda$decode$0(WebRTCVideoForwarder.java:184)
        at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313)
        at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NullPointerException
        at ai.djl.pytorch.jni.LibUtils.findNativeLibrary(LibUtils.java:286)
        at ai.djl.pytorch.jni.LibUtils.getLibName(LibUtils.java:100)
        at ai.djl.pytorch.jni.LibUtils.loadLibrary(LibUtils.java:88)
        at ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:50)
        ... 22 more

I'm not buiding my project with gradle but with maven. I don't know if that could be a problem..

I'm a bit lost. I'll try to publish my project on github but it's quite a mess to test it!

[frankfliu]

@midgard-as

This looks like a bug in DJL, for some reason your current thread's context classloader is null. I create a PR to address this issue:

#1445

[midgard-as]

Hi,
Thank you for the info. I saw that the PR has been merged already, thank you very much. I managed to build last jars with previous command line and add them to my project and I now have a new error that i don't really understand :

Loading: 100% |████████████████████████████████████████| error on model load ai.djl.engine.EngineException: Failed to load PyTorch native libraryai.djl.engine.EngineException: Failed to load PyTorch native library at ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:77) at ai.djl.pytorch.engine.PtEngineProvider.getEngine(PtEngineProvider.java:40) at ai.djl.engine.Engine.getEngine(Engine.java:177) at ai.djl.Model.newInstance(Model.java:98) at ai.djl.repository.zoo.BaseModelLoader.createModel(BaseModelLoader.java:187) at ai.djl.repository.zoo.BaseModelLoader.loadModel(BaseModelLoader.java:149) at ai.djl.repository.zoo.Criteria.loadModel(Criteria.java:166) at io.antmedia.filter.FaceBlur.<init>(FaceBlur.java:116) at io.antmedia.filter.FaceBlur.getInstance(FaceBlur.java:129) at io.antmedia.filter.BlurAdaptor.getFrameFilter(BlurAdaptor.java:308) at io.antmedia.filter.BlurAdaptor.update(BlurAdaptor.java:166) at io.antmedia.filter.BlurAdaptor.onVideoFrame(BlurAdaptor.java:81) at io.antmedia.filter.BlurAdaptor.onVideoPacket(BlurAdaptor.java:283) at io.antmedia.plugin.PacketFeeder.writePacket(PacketFeeder.java:43) at io.antmedia.plugin.PacketFeeder.writeVideoBuffer(PacketFeeder.java:95) at io.antmedia.enterprise.adaptive.WebRTCVideoForwarder.writeVideoBuffer(WebRTCVideoForwarder.java:93) at io.antmedia.enterprise.adaptive.WebRTCVideoForwarder$VirtualVideoDecoder.lambda$decode$0(WebRTCVideoForwarder.java:184) at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.UnsatisfiedLinkError: /usr/local/antmedia/lib/native/libgtest_main.a: /usr/local/antmedia/lib/native/libgtest_main.a: invalid ELF header (Possible cause: endianness mismatch) at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method) at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2442) at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2498) at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694) at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627) at java.base/java.lang.Runtime.load0(Runtime.java:768) at java.base/java.lang.System.load(System.java:1837) at ai.djl.pytorch.jni.LibUtils.loadNativeLibrary(LibUtils.java:319) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at ai.djl.pytorch.jni.LibUtils.loadLibTorch(LibUtils.java:120) at ai.djl.pytorch.jni.LibUtils.loadLibrary(LibUtils.java:72) at ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:50) ... 22 more

Do you have any idea what this could be due to ?

Thanks !

[frankfliu]

@midgard-as

Can you try our nightly snapshot build?
See pom.xml example: https://github.com/deepjavalibrary/djl/blob/master/examples/pom.xml#L17-L34

[midgard-as]

Thank you very much for your help, it worked with the snapshot ! My last issue was unrelated to DJL :)