How to provide "serialized_example" input

[kaicao]

Hi,
I've trained a model based on tensorflow/models (with 2.7.0 TensorFlow) ssd_mobilenet_v2_coco.config
By showing the signature of it, it requires one STRING typed input for "serialized_sample":

The given SavedModel SignatureDef contains the following input(s):
  inputs['serialized_example'] tensor_info:
      dtype: DT_STRING
      shape: ()
      name: tf_example:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['detection_boxes'] tensor_info:
...

Thus when I used the DJL (version 0.14.0) to load the model and use SingleShotDetetionTranslator for prediction, and providing Image like
Image image = BufferedImageFactory.getInstance().fromFile(file);
It ends up error when predict on the image
Expects arg[0] to be string but float is provided.

Any advice how could it work?

[frankfliu]

@kaicao
TensorFlow DT_STRING data type usually is used to represent raw bytes. In this case you should create your own Translator. You can create String NDArray like the following:

    NDList processInput(TranslatorContext ctx, Path imageFile) throws Exception;
        byte[] input = Files.readAllbytes(imageFile);
        TfNDManager manager = ((TfNDManager)ctx.getNDManager());
        NDArray array = manager.createStringTensor(new Shape(1), ByteBuffer.wrap(input));
        array.setName("tf_example:0");
        return new NDList(array);
    }

[kaicao]

Thank you for your answer, I created new Translator based on your suggested code.
There is error when executing predictor with that translator, hope you could help:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000013a234ff1, pid=71137, tid=0x0000000000002703
#
# JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C  [libtensorflow_cc.2.dylib+0xb510ff1]  _ZN10tensorflow17TF_TensorToTensorEPK9TF_TensorPNS_6TensorE+0x11
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /.../hs_err_pid71137.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

By looking into error report there is more detailed stack:

Stack: [0x00007000014f6000,0x00007000015f6000],  sp=0x00007000015f4eb0,  free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libtensorflow_cc.2.dylib+0xb510ff1]  _ZN10tensorflow17TF_TensorToTensorEPK9TF_TensorPNS_6TensorE+0x11
C  [libtensorflow_cc.2.dylib+0x1162e]  _ZL13TF_Run_InputsPKP9TF_TensorPNSt3__16vectorINS3_4pairINS3_12basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEEN10tensorflow6
TensorEEENS9_ISE_EEEEP9TF_Status+0x8e
C  [libtensorflow_cc.2.dylib+0x2157d]  TF_SessionRun+0x14d
C  [libjnitensorflow.dylib+0x2c6e6]  Java_org_tensorflow_internal_c_1api_global_tensorflow_TF_1SessionRun__Lorg_tensorflow_internal_c_1api_TF_1Session_2Lorg_tensorflow_int
ernal_c_1api_TF_1Buffer_2Lorg_tensorflow_internal_c_1api_TF_1Output_2Lorg_bytedeco_javacpp_PointerPointer_2ILorg_tensorflow_internal_c_1api_TF_1Output_2Lorg_bytedeco_javac
pp_PointerPointer_2ILorg_bytedeco_javacpp_PointerPointer_2ILorg_tensorflow_internal_c_1api_TF_1Buffer_2Lorg_tensorflow_internal_c_1api_TF_1Status_2+0x356
j  org.tensorflow.internal.c_api.global.tensorflow.TF_SessionRun(Lorg/tensorflow/internal/c_api/TF_Session;Lorg/tensorflow/internal/c_api/TF_Buffer;Lorg/tensorflow/interna
l/c_api/TF_Output;Lorg/bytedeco/javacpp/PointerPointer;ILorg/tensorflow/internal/c_api/TF_Output;Lorg/bytedeco/javacpp/PointerPointer;ILorg/bytedeco/javacpp/PointerPointer
;ILorg/tensorflow/internal/c_api/TF_Buffer;Lorg/tensorflow/internal/c_api/TF_Status;)V+0
j  ai.djl.tensorflow.engine.javacpp.JavacppUtils.runSession(Lorg/tensorflow/internal/c_api/TF_Session;Lorg/tensorflow/proto/framework/RunOptions;[Lorg/tensorflow/internal/
c_api/TF_Tensor;[Lorg/tensorflow/internal/c_api/TF_Operation;[I[Lorg/tensorflow/internal/c_api/TF_Operation;[I[Lorg/tensorflow/internal/c_api/TF_Operation;)[Lorg/tensorflow/internal/c_api/TF_Tensor;+277
j  ai.djl.tensorflow.engine.TfSymbolBlock.forwardInternal(Lai/djl/training/ParameterStore;Lai/djl/ndarray/NDList;ZLai/djl/util/PairList;)Lai/djl/ndarray/NDList;+202
j  ai.djl.nn.AbstractBlock.forward(Lai/djl/training/ParameterStore;Lai/djl/ndarray/NDList;ZLai/djl/util/PairList;)Lai/djl/ndarray/NDList;+36
j  ai.djl.nn.Block.forward(Lai/djl/training/ParameterStore;Lai/djl/ndarray/NDList;Z)Lai/djl/ndarray/NDList;+5
j  ai.djl.inference.Predictor.predictInternal(Lai/djl/translate/TranslatorContext;Lai/djl/ndarray/NDList;)Lai/djl/ndarray/NDList;+21
j  ai.djl.inference.Predictor.batchPredict(Ljava/util/List;)Ljava/util/List;+245
j  ai.djl.inference.Predictor.predict(Ljava/lang/Object;)Ljava/lang/Object;+5

[frankfliu]

@kaicao
Do you have a github project that can reproduce this crash?

[kaicao]

Committed the codes to https://github.com/kaicao/PikachuDetector
And this is the main class that I used to hit on the issue DJLPikachuDetectHandler

[frankfliu]

@kaicao

Your input seems to be a scalar, the shape should be new Shape(), and the name should be serialized_example:

    NDArray array = manager.createStringTensor(new Shape(), ByteBuffer.wrap(input));
    array.setName("serialized_example");

But it looks like your model is not able to read image file, do you have python code that can run inference of your model?
Or do you have a working version of inference using TensorFlow java directly?

[kaicao]

Found solution here.

Cause of issue: The model was exported with tf_example, which requires serialized TFExample protos as mentioned in export_inference_graph.py.

It's bit tricky for me to get TFExample protos serialization to work by using tensorflow-core library.
So instead have solution:

1. exported with new input of encoded_image_string_tensor, that accepts 1-D string tensor containing encoded PNG or JPEG images.
2. Execute Java application with the new exported model by feeding encoded_image_string_tensor with image.

Thank you for your help along the way @frankfliu !