Assertion `mat->info->is_wavefunction' failed.

[cfengno1]

Hi Huanchen,

My large-scale dmrg calculations was killed due to the time-limit of the cluster. When I try to restart the calculation from the saved MPS I encountered error:

    456 GB <-- Site =  118 .. Mmps =16000 Ndav =  16 E =   -619.1563107987 Error = 1.69e-11 FLOPS = 2.75e+11 Tdav = 856.42 T = 1126.65 
    456 GB <-- Site =  117 .. Mmps =16000 Ndav =  11 E =   -619.1564308219 Error = 1.79e-11 FLOPS = 3.01e+11 Tdav = 526.16 T = 777.92
    456 GB <-- Site =  116   

**killed due to time limit**

**Restart...**

Sweep =   12 | Direction = backward | Bond dimension = 16000 | Noise =  0.00e+00 | Dav threshold =  1.00e-06
    310 GB <-- Site =  116 .. python: /usw/u/bbe3871/block2/src/core/sparse_matrix.hpp:1870: void block2::SparseMatrix<S, FL>::swap_to_fused_right(const std::shared_ptr<block2::SparseMatrix<S, FL> >&, const block2::StateInfo<S>&, const block2::StateInfo<S>&, const block2::StateInfo<S>&, const block2::StateInfo<S>&, const std::shared_ptr<typename block2::StateInfo<S>::ConnectionInfo>&, const block2::StateInfo<S>&, const std::shared_ptr<typename block2::StateInfo<S>::ConnectionInfo>&, const std::shared_ptr<block2::CG<S> >&) [with S = block2::SAnyT<>; FL = std::complex<double>; typename block2::StateInfo<S>::ConnectionInfo = block2::StateInfo<block2::SAnyT<>, void>::ConnectionInfo]: Assertion `mat->info->is_wavefunction' failed.

When I do "ls -l" to check the update time of the MPS files:

-rw-r--r-- 1 bbe3871 bbe3871      2012 Jan  1 13:19 F.MPS.INFO.KET.LEFT.117
-rw-r--r-- 1 bbe3871 bbe3871      2012 Jan  4 14:56 F.MPS.INFO.KET.RIGHT.117
-rw-r--r-- 1 bbe3871 bbe3871 750371746 Jan  4 14:56 F.MPS.KET.117
-rw-r--r-- 1 bbe3871 bbe3871      1932 Jan  1 13:02 F.MPS.INFO.KET.LEFT.116
-rw-r--r-- 1 bbe3871 bbe3871      1932 Jan  4 15:16 F.MPS.INFO.KET.RIGHT.116
-rw-r--r-- 1 bbe3871 bbe3871 741440386 Jan  4 15:16 F.MPS.KET.116
-rw-r--r-- 1 bbe3871 bbe3871      1892 Jan  1 12:39 F.MPS.INFO.KET.LEFT.115
-rw-r--r-- 1 bbe3871 bbe3871      1892 Dec 29 16:10 F.MPS.INFO.KET.RIGHT.115
-rw-r--r-- 1 bbe3871 bbe3871 742782594 Jan  1 13:02 F.MPS.KET.115

it seems that the program was killed before MPS at 115th site could be saved.

Is there any way to fix this error? Or must I use a previously saved MPS? How could I prevent this error to happen?

Best,
Feng

[hczhai]

There are certain rules for restarting, for example, "for the MPS loaded from the interrupted sweep, driver.adjust_mps should not be used after driver.load_mps". However, there is no guarantee that you can always restart from the middle of a sweep, since the job can be killed when the file saving is done partially and the file is broken. If you use driver.adjust_mps incorrectly it can also cause the file to become broken. For more reliable way of restarting large jobs, you need to use the restart_dir argument (see https://block2.readthedocs.io/en/latest/tutorial/restarting-dmrg.html#Restarting-from-backup-MPS), and restart from the backup MPS when loading from the interrupted sweep is not possible.