Perplexed by the indeterminacy of the Model Using GIN as an Encoder with GPU

[ZJUDataIntelligence]

Recently, I have encountered some perplexing issues while using GIN as an encoder. I have found that I cannot reproduce previous results, even when using the same data, parameters, and the same random seed. I have tried various methods to address this problem, including checking the code and environment settings, but have not found any apparent reasons. This irreproducibility has left me feeling frustrated as I cannot comprehend why the same model produces different results across different runs.

Later, I read the documentation of PYG and came across a mention of "Memory-Efficient Aggregations" where it states, "As an additional advantage, MessagePassing implementations that utilize the SparseTensor class are deterministic on the GPU since aggregations no longer rely on atomic operations." Does this uncertainty arise from atomic operations? However, I also attempted to run the SparseTensor version of GIN and found that the results still varied, which has further deepened my confusion.

[rusty1s]

Yes, atomic operations are non-deterministic in the sense that the order of aggregation is undefined, which may lead to slightly different results to how floating point precision works. This is indeed resolved by the usage of SparseTensor (which uses a different reduction scheme on the GPU). I am not able to reproduce your non-determinism issues with SparseTensor, so this might come from something else, e.g., a global_add_pool call inside your model.

[ZJUDataIntelligence]

Thanks for your answer! I did use global_add_pool in my model. Is there randomness in this operation? I am directly using Pyg's built-in pool method. Also, is there a more detailed explanation of the randomness that occurs when atomic operations result in a different order or do you have any profile or code you would like to share with me? I hope to have a deeper understanding, thank you very much!

[rusty1s]

All scatter operations are non-deterministic, which are used within neighborhood aggregation and node-level aggregation. You can use torch_scatter.segment_csr from torch-scatter for deterministic aggregation in readout:

from torch_scatter import segment_csr

# Replace
out = global_add_pool(x, data.batch)
# with
out = segment_csr(x, data.ptr, reduce="sum")