Having Trouble w/ Step 3 (using run_step3_sd_nacs_libint) Generating the Imaginary res-sd Files

[joan0j]

Hi there! To provide some context, I'm an undergraduate student who started programming this year and began an internship running simulations using libra-code. I've had to learn a lot but it's been enjoyable! That being said... I have hit my first hurdle I haven't been able to get over.

I have been able to successfully run step 3 using the function 'run_step3_sd_nacs_libint', producing the expected files in the res-sd and res-ks directory. However, when I have been running this function in a python file it hasn't been producing the expected outputs in res-sd. (My lastest run produced the real wavefunctions of for /res-sd/ but not the imaginary)

Trial 1:
I ran this without altering the /res/ and /all_logfiles/ .npz files output from step 2, and it only generated the res-ks directory without giving any errors or messages in the log.

Trial 2:
I wrote a python function using numpy to help process the npz files, which I will copy a snippet of the output below, and it did create the real-wavefunction npz files for res-sd but not the imaginary.
Snippet of the .output
Processing key 'indices' with original type: <class 'numpy.ndarray'>
Processing key 'indptr' with original type: <class 'numpy.ndarray'>
Processing key 'format' with original type: <class 'numpy.ndarray'>
Cannot convert key 'format' to numeric type. Setting to zeros.
Processing key 'shape' with original type: <class 'numpy.ndarray'>
Processing key 'data' with original type: <class 'numpy.ndarray'>
Converted data saved to /srv/step2/res/S_ks_5689_converted.npz.

Trial 3:
I reran from the beginning incase my files got corrupted somewhere along the way, starting with the optimization. I ran up to step 3 again and still- no res-sd directory produced.
Using the python function I wrote to view the .npz files, I was able to get the real npz files but again not the imaginary.

I will also attach the files I am submitting for this job, which I am running on a supercluster with condor. Please let me know if you have any ideas or suggestions, or questions to clarify.

Any help is appreciated :)
debug_sh.txt
step3_python.txt

[alexvakimov]

@MohammadShakiba , could you please take a look into it when you have a moment?

[MohammadShakiba]

Hi @joan0j,
I've checked your snippet and you are saving the npz files in a different way. What we do in step 3 is to make them as sparse format and save them in a column-major format. You need just a couple of lines to do that with the scipy.sparse library as follows:

import scipy.sparse as sp
## First part on loading the npz files
sparse_mat = sp.load_npz('/path/to/file.npz')
# You can extract it via the following
# This will give you an numpy.ndarray which contains complex numbers which are 0.0
# Note that the real and imaginary parts are saved with different naming in Libra like Hvib_sd_10_re.npz (which are the energies)
# and Hvib_sd_10_im.npz (which are the couplings)
dense_mat = sparse_mat.todense() # You can extract the real part by
dense_mat = dense_mat.real
## Second part on how to save an npz file using scipy.sparse
# I imagine you have a numpy array called `test_array`
test_sparse = sp.csc_matrix(test_array)
sp.save_npz('/path/to/savefile/file.npz', test_sparse)

What you have shown in your snippet is a complex way of loading npz files with numpy which I believe may not be compatible with the way they were saved using scipy.sparse.save_npz function.