ASR in Unreal Engine C++

[DominicHughes899]

Hi,

I'm trying to get ASR working inside Unreal Engine for a university project. Does anyone know if this would be possible using sherpa, or even if this is the best direction for me to take?

I don't know if I need to build libs or if I can do this another way, any help would be greatly appreciated :)

[csukuangfj]

yes, it's definitely possible.

You can either download pre-built libs from us or you can build sherpa-onnx from source.

[csukuangfj]

You can invoke the "Release()" method of the stream and use C API to free the returned pointer from Release()

[DominicHughes899]

This might be silly, but is there a way to orphan the offline stream object (allow memory leak) as a proof of concept?

[csukuangfj]

Yes, as commented above. Use Release() without calling the c api.

[DominicHughes899]

Okay, I'm gonna mark this as solved.

Unreal doesn't like the cxx api because of the MoveOnly Class, I think it clashes with Unreals garbage collection system.

I hate that I didn't try this earlier but I've used c-api instead (following closely to c api example), raw c code amongst a cpp class and it works perfectly!

Thanks Fangjun Kuang! Your help is greatly appreciated, my gratitude is immeasurable.
(I will probably be back with more questions soon... :D)

[csukuangfj]

When I use offlineRecognizerResult, unreal sometimes throws an exception because both unreal and the sherpa have separate new/delete overrides (I think). Does this sound correct?

I think you are right.

We should not return a struct object containing std::vector or std::string to UE. Otherwise, the memory is allocated in sherpa-onnx using an allocator from c++ standard library, but it is freed by UE using its own memory allocator.

#2349 is an example fix for using tts in UE. Someone is helping test it in UE. If it works, we can do the same thing for ASR.