FeatureStore and get_tensor high latency: workarounds?

[denadai2]

Dear PyG community,
I am implementing a custom FeatureStore. However, my get_tensor operations have some inevitable latency due to my cluster. Is it possible to have some workaround to avoid slowing down the training?

I tried increasing the number of workers/prefetching but this does not seem to help.

thanks

[rusty1s]

To which database does get_tensor() communicate with? In the end, this is hard to tell. If num_workers > 0 doesn't help, it may means that your get_tensor() implementation cannot handle multiple accesses in parallel but accesses features sequentially.

[denadai2]

I'm using Ray, which supports parallel gets. However, it seems calls to get_tensor are done almost sequentially (from the logs). Can it be?

[rusty1s]

get_tensor is called sequentially for different attributes in your graph, yes. If you want to parallelize that, you can also implement multi_get_tensor. In data loading, get_tensor() should be then called in each worker process.