permute_ destroys data structure of UniTensor with shared blocks

[manuschneider]

If two 'UniTensor's have the same blocks, and I use permute_ on one of them, then the data structure of the other UniTensor is not consistent with its metadata anymore.

Here is an example that looks innocent enough:

uT = cytnx.UniTensor.arange(6).reshape_(2,3).set_name("uT").set_rowrank(1)
print(uT) #shows Tensor with shape (2,3)
uT2 = uT.relabel(["a","b"]).set_name("uT2").permute_([1,0]).set_name("uT2")
uT2[0,1] = 9
uT.print_diagram() #has shape (2,3)
print(uT) #shows Tensor with shape (3,2)!!!
uT2.print_diagram() #has shape (3,2)
print(uT2) #shows Tensor with shape (3,2)

Output:

-------- start of print ---------
Tensor name: uT
Tensor type: Dense
is_diag    : False
contiguous : True

Total elem: 6
type  : Double (Float64)
cytnx device: CPU
Shape : (2,3)
[[0.00000e+00 1.00000e+00 2.00000e+00 ]
 [3.00000e+00 4.00000e+00 5.00000e+00 ]]




-----------------------
tensor Name : uT
tensor Rank : 2
block_form  : False
is_diag     : False
on device   : cytnx device: CPU
          ---------     
         /         \    
   0 ____| 2     3 |____ 1
         \         /
          ---------     
-------- start of print ---------
Tensor name: uT
Tensor type: Dense
is_diag    : False
contiguous : False

Total elem: 6
type  : Double (Float64)
cytnx device: CPU
Shape : (3,2)
[[0.00000e+00 9.00000e+00 ]
 [1.00000e+00 4.00000e+00 ]
 [2.00000e+00 5.00000e+00 ]]




-----------------------
tensor Name : uT2
tensor Rank : 2
block_form  : False
is_diag     : False
on device   : cytnx device: CPU
          ---------     
         /         \    
   b ____| 3     2 |____ a
         \         /
          ---------     
-------- start of print ---------
Tensor name: uT2
Tensor type: Dense
is_diag    : False
contiguous : False

Total elem: 6
type  : Double (Float64)
cytnx device: CPU
Shape : (3,2)
[[0.00000e+00 9.00000e+00 ]
 [1.00000e+00 4.00000e+00 ]
 [2.00000e+00 5.00000e+00 ]]

This leads to errors that are difficult to find.

I suggest that UniTensor.permute_() should call Tensor.permute() internally for each block. The metadata of the Tensor is copied in this case, but the data in memory remains untouched, so the overhead is not large.

[manuschneider]

reshape_() is similarly affected, as can be seen by replacing permute_([1,0]) by reshape_(3,2) in the above example.

[IvanaGyro]

This is an old topic related to the copy-view design. You can refer the discussion in Python array API standard.

Here is a quote from the document which briefly describes the complexity to "solve" the problem about shared blocks (views).

A potential solution could be to make views read-only or implement copy-on-write semantics. Both are hard to implement and would present significant backward compatibility issues for current strided array libraries. Read-only views would also not be a full solution due to the fact that mutating the original (base) array will also result in ambiguous semantics. Accordingly, this standard does not attempt to pursue this solution.

In Cytnx, Tensor and UniTensor are both view-like components.

[ianmccul]

Also relevant here is that with non-abelian symmetries (or more generally, non-trivial fusion), permuting indices of a tensor is a more complicated operation that requires phase factors and/or normalization factors on each block, or even, F-moves (non-trivial resummation over internal indices). So its not something you can easily take a 'view' of without affecting the underlying data, at least in the more complicated cases.

[manuschneider]

In Cytnx, Tensor and UniTensor are both view-like components.

I agree, there is no solution that fits it all. For example, if I get a tensor with get_block and manipulate it, I will always corrupt the original UniTensor. However, I think it is possible to have a decent behavior for most users, while assuming that those dealing with blocks and internal structure of UniTensors know what they are doing.

We luckily have some abstraction layers that we can use: UniTensor -> Tensor -> Storage.
If a UniTensor is permuted, for example, we can call Tensor.permute() on the blocks instead of Tensor.permute_(). This way, the metadata of the block is changed without affecting other views of that Tensor.
The cost is that the metadata of the Tensor has to be created newly, which in principle causes a tiny amount of memory overhead. But then, permute changes basically all this metadata anyways and it should not matter performance-wise if we copy or overwrite.

What remains is that the Storage is shared between different UniTensors / Tensors, as addressed in the link above. But this is what we want, because we do not want to copy the data all the time.

We do the behavior that I suggested already elsewhere: Tensor.contiguous_() internally calls Storage.Move_memory(), not the inline version with underscore! If two tensors refer to the same memory and one is permuted followed by contiguous(), then the other tensor is not corrupted by this.
I think we should do the same for UniTensors by calling Tensor.permute() and Tensor.reshape() instead of the inline versions.

Also relevant here is that with non-abelian symmetries (or more generally, non-trivial fusion)

I think these cases can be dealt with correctly: since the data has to be changed when permuting, we should create a new Tensor and Storage anyways. This means that other views are not affected.

[ianmccul]

Right, being able to reuse some blocks while recomputing some others, gives a lot of flexibility and should be quite efficient. The cost of copying the metadata is not likely to be relevant.