NNAPI flag ignored

[marcinkeviciusp]

I've been trying to benchmark NNAPI as compared to CPU and some other frameworks, but whenever I try to make it use NNAPI (options.addNnapi(flags) in Java or OrtSessionOptionsAppendExecutionProvider_Nnapi(session_options, nnapi_flags) in C++) the performance stays identical to CPU - in fact, changing number of of threads or any other parameters (e.g. batch size, quantization) affects the resulting time identically as well - this leads me to believe that it's still just using / falling back on CPU.

In the Java project I used implementation 'com.microsoft.onnxruntime:onnxruntime-mobile:1.8.2', while for C++ I built the library (with --use_nnapi) myself.

Is there any way I can debug this (I've not found any ways to output which EP is actually used)? Or perhaps there's some known issue I'm not aware of? It seems like some combination of all these attempts should have worked...

I have tested with:

• Model: ResNet50 (quantized and unquantized; .onnx and .ort (converted with and without --use_nnapi flag - not using the flag does make it crash on inference telling to use the flag))
• Device: Samsung S20 (which, according to my research, has an NPU, which NNAPI should utilize).

Edit: I should also mention I've tried all NNAPI flag coombinations (enable fp16, disable cpu, nchw) which had no effect.

[edgchen1]

There are some log messages that may be useful. You may need to set the log level to a more verbose level to see them.

Statistics about how many nodes the NNAPI EP supports (INFO log level):

onnxruntime/onnxruntime/core/providers/nnapi/nnapi_builtin/nnapi_execution_provider.cc

Lines 200 to 212 in 430e80e

	const auto summary_msg = MakeString(
	"NnapiExecutionProvider::GetCapability,",
	" number of partitions supported by NNAPI: ", num_of_partitions,
	" number of nodes in the graph: ", graph_view.NumberOfNodes(),
	" number of nodes supported by NNAPI: ", num_of_supported_nodes);
	// If the graph is partitioned in multiple subgraphs, and this may impact performance,
	// we want to give users a summary message at warning level.
	if (num_of_partitions > 1) {
	LOGS_DEFAULT(WARNING) << summary_msg;
	} else {
	LOGS_DEFAULT(INFO) << summary_msg;
	}

Information about node EP assignment (VERBOSE log level):

onnxruntime/onnxruntime/core/session/inference_session.cc

Lines 908 to 920 in 430e80e

	if (is_verbose_mode) {
	LOGS(*session_logger_, VERBOSE) << "Node placements";
	if (node_placements.size() == 1) {
	LOGS(*session_logger_, VERBOSE) << "All nodes have been placed on [" << node_placements.begin()->first << "].";
	} else {
	for (const auto& pr : node_placements) {
	std::ostringstream all_nodes_str;
	std::copy(pr.second.begin(), pr.second.end(), std::ostream_iterator<std::string>(all_nodes_str, ", "));
	LOGS(*session_logger_, VERBOSE) << " Provider: [" << pr.first << "]"
	<< ": [" << all_nodes_str.str() << "]";
	}
	}
	}

[guoyu-wang]

ResNet50 is support by our NNAPI execution provider, and can take advantage of the hardware accelerators in Samsung s20.
However, NNAPI execution provider does not support models with dynamic input shapes, this is probably the reason that none of the NNAPI options worked since the execution always falls back to CPU execution provider.
Please try the same ResNet50 model with static input shape.

[vpenades]

I am having similar behavior, my model's performance is the same regardless of the flags I use; input is fixed size and NCHW

It could be good to have a general "tips & tricks" to optimize performance on Android/NNAPI