[neural.slang] Remove the usage of pointer to shared memory

[kaizhangNV]

Problem Description

We need to remove the usage of pointer to shared memory. Because it's very easy to trigger bug in the downstream driver on Vulkan.

Preferred Solution

We will directly use the shared memory array.