Double/Float error deploying pair_allegro

[apoletayev]

Hi folks! I've been trying to use pair_allegro with NPT simulations. I am getting a float vs double error which I'm guessing has to do with either the LAMMPS version, or my building LAMMPS, or the model's float/double configuration.

nequip@develop (0.6.0) commit 8310052ccc022fe4ddf8231933f037c525051a5c
allegro@main commit 71ea80992f70a84fc65ecfb5a5ca45e8a82c6031
pair_allegro@stress (all following this comment) commit 176db81
for both training and building LAMMPS: CUDA 11.3.1, libtorch 1.11.0
LAMMPS 28Mar2023 (latest) patched with pair_allegro and built with libtorch + kokkos, tested without pair_allegro.

Posisbly relevant parts of the pytorch training config:

default_dtype: float64
model_dtype: float32

LAMMPS call : lmp -sf kk -k on gpus 1 -pk kokkos newton on neigh full -in in.LPS -var structure structure.lmp -var model model.pth

LAMMPS input file :

units metal              # Check this! It must match the units of the training data + NN

dimension 3
boundary p p p
atom_style atomic        # atomic or charge can be the output of atomsk

newton on                

read_data ${structure}     ## read the structure from command-line parameter

group lithium type 1       ## define species 1 as lithium.
group phospho type 2       ## define species 2 as phosphorus.
group sulphur type 3       ## define species 3 as S

pair_style hybrid/overlay lj/cut 0.5 allegro        # short-range LJ repulsion with Allegro.
pair_coeff * * lj/cut 1 0.5                         # short-range LJ repulsion
pair_coeff * * allegro ${model} Li P S

min_style cg                                                # minimization style - conjugate gradient
minimize 1e-25 1e-25 5000 10000           # this fails

Error trace from LAMMPS:

LAMMPS (28 Mar 2023 - Development)
KOKKOS mode is enabled (src/KOKKOS/kokkos.cpp:106)
  will use up to 1 GPU(s) per node
Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set
  In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads
  For best performance with OpenMP 3.1 set OMP_PROC_BIND=true
  For unit testing set OMP_PROC_BIND=false
  using 1 OpenMP thread(s) per MPI task
Reading data file ...
  triclinic box = (0 0 0) to (26.107018 15.872317 12.23866) with tilt (-0.065437871 0.059841395 -0.097553592)
  1 by 1 by 1 MPI processor grid
  reading atoms ...
  256 atoms
  read_data CPU = 0.006 seconds
96 atoms in group lithium
32 atoms in group phospho
128 atoms in group sulphur
Allegro is using input precision f and output precision d
Allegro is using device cuda
Allegro: Loading model from stress_LPS_B6_l1_064_all.pth
Allegro: Freezing TorchScript model...
Type mapping:
Allegro type | Allegro name | LAMMPS type | LAMMPS name
0 | Li | 1 | Li
1 | P | 2 | P
2 | S | 3 | S
Neighbor list info ...
  update: every = 1 steps, delay = 0 steps, check = yes
  max neighbors/atom: 2000, page size: 100000
  master list distance cutoff = 8
  ghost atom cutoff = 8
  binsize = 8, bins = 4 2 2
  2 neighbor lists, perpetual/occasional/extra = 2 0 0
  (1) pair lj/cut, perpetual, trim from (2)
      attributes: full, newton on, kokkos_device, cut 2.5
      pair build: trim/kk/device
      stencil: none
      bin: none
  (2) pair allegro, perpetual
      attributes: full, newton on, ghost, kokkos_device
      pair build: full/bin/ghost/kk/device
      stencil: full/ghost/bin/3d
      bin: kk/device
Setting up cg/kk style minimization ...
  Unit style    : metal
  Current step  : 0
WARNING: Fix with atom-based arrays not compatible with sending data in Kokkos communication, switching to classic exchange/border communication (src/KOKKOS/comm_kokkos.cpp:666)
terminate called after throwing an instance of 'c10::Error'
  what():  expected scalar type Double but found Float
Exception raised from data_ptr<double> at aten/src/ATen/core/TensorMethods.cpp:18 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x1536d7d490eb in /path/to/libtorch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x1536d7d44abe in /path/to/libtorch/lib/libc10.so)
frame #2: double* at::TensorBase::data_ptr<double>() const + 0x10b (0x15363b8be32b in /path/to/libtorch/lib/libtorch_cpu.so)
frame #3: LAMMPS_NS::PairAllegroKokkos<(Precision)2>::compute(int, int) + 0x2ddb (0x1535f59b0d5b in /path/to/liblammps.so.0)
frame #4: LAMMPS_NS::PairHybridKokkos::compute(int, int) + 0x5bb (0x1535f5b6ad7b in /path/to/liblammps.so.0)
frame #5: LAMMPS_NS::MinKokkos::setup(int) + 0xaba (0x1535f4fdc62a in /path/to/liblammps.so.0)
frame #6: LAMMPS_NS::Minimize::command(int, char**) + 0x1bb (0x1535f3fe4e6b in /path/to/liblammps.so.0)
frame #7: LAMMPS_NS::Input::execute_command() + 0xbe3 (0x1535f3f666d3 in /path/to/liblammps.so.0)
frame #8: LAMMPS_NS::Input::file() + 0x155 (0x1535f3f66955 in /path/to/liblammps.so.0)
frame #9: /path/to/lmp() [0x4010ea]
frame #10: __libc_start_main + 0xf3 (0x1535f28c0873 in /lib64/libc.so.6)
frame #11: /path/to/lmp() [0x40119e]

 *** Process received signal ***
 Signal: Aborted (6)
 Signal code:  (-6)
 [ 0] /path/to/lib64/libc.so.6(+0x37960)[0x1535f28d4960]
 [ 1] /path/to/lib64/libc.so.6(gsignal+0x10f)[0x1535f28d48df]
 [ 2] /path/to/lib64/libc.so.6(abort+0x127)[0x1535f28becf5]
 [ 3] /path/to/lib64/libstdc++.so.6(+0xad9c3)[0x1535f2d0d9c3]
 [ 4] /path/to/lib64/libstdc++.so.6(+0xb92b6)[0x1535f2d192b6]
 [ 5] /path/to/lib64/libstdc++.so.6(+0xb9321)[0x1535f2d19321]
 [ 6] /path/to/lib64/libstdc++.so.6(+0xb9575)[0x1535f2d19575]
 [ 7] /path/to/libtorch/lib/libc10.so(_ZN3c106detail14torchCheckFailEPKcS2_jRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0xfa)[0x1536d7d44aea]
 [ 8] /path/to/libtorch/lib/libtorch_cpu.so(_ZNK2at10TensorBase8data_ptrIdEEPT_v+0x10b)[0x15363b8be32b]
 [ 9] /path/to/liblammps.so.0(_ZN9LAMMPS_NS17PairAllegroKokkosIL9Precision2EE7computeEii+0x2ddb)[0x1535f59b0d5b]
 [10] /path/to/liblammps.so.0(_ZN9LAMMPS_NS16PairHybridKokkos7computeEii+0x5bb)[0x1535f5b6ad7b]
 [11] /path/to/liblammps.so.0(_ZN9LAMMPS_NS9MinKokkos5setupEi+0xaba)[0x1535f4fdc62a]
 [12] /path/to/liblammps.so.0(_ZN9LAMMPS_NS8Minimize7commandEiPPc+0x1bb)[0x1535f3fe4e6b]
 [13] /path/to/liblammps.so.0(_ZN9LAMMPS_NS5Input15execute_commandEv+0xbe3)[0x1535f3f666d3]
 [14] /path/to/liblammps.so.0(_ZN9LAMMPS_NS5Input4fileEv+0x155)[0x1535f3f66955]
 [15] /path/to/lmp[0x4010ea]
 [16] /path/to/lib64/libc.so.6(__libc_start_main+0xf3)[0x1535f28c0873]
 [17] /path/to/lmp[0x40119e]
 *** End of error message ***

Very nearly same or exactly the same error shows up if I use a model trained without stress with nequip 0.5.5 having default_dtype: float32 calling it as allegro3232 from LAMMPS. Building 29Sep2021_update1 release of lammps patched with pair_allegro@master works.

I would really appreciate any suggestions around what I could tune to test/eliminate this. Currently testing other releases of LAMMPS and commits of the stress branch. Hopefully there is a combination of both that should work.

[Linux-cpp-lisp]

Hi @apoletayev ,

Hm, at least from looking over this now everything looks correct... the versions and precisions match up. My first guess is that one of the versions isn't what you think it is (pip install vs pip install -e, confirm ./patch_lammps, delete and use clean build/ dir?) but that might not be it. I've been using almost exactly this set of commits through the last two weeks so something is off.

You could try to run the unit tests on pair_allegro using pytest. Just be sure to be in the conda env you use for training; you can provide as long a command as you need in the LAMMPS env var to pytest for running the LAMMPS you've built, including things like conda run to run LAMMPS in a different environment than runs the Python side of the tests.

As far as I know we've never tested pair_style hybrid/overlay on the LAMMPS side with this, but I don't see how that could yield this error.

[apoletayev]

Thank you, will test some more. I have been experimenting with versions of lammps and commits of pair_allegro, so everything about those two is re-downloaded afresh every try.
The remaining thing I can think of is that the conda env which trained the deployed models has the full pytorch and cudatoolkit (conda install -c pytorch pytorch=1.11.0 cudatoolkit=11.3.1), but not libtorch or cuDNN. I do not know how I would train using libtorch, or install nequip/allegro without pytorch in the environment. Those are new for lammps because I wanted to build lammps with kokkos. The versions of CUDA and pytorch are same for training and lammps. So I'll test installing lammps without libtorch or kokkos, and the other way around, removing pytorch / cudatoolkit from the environment and only leaving torch bindings.

[Linux-cpp-lisp]

Hm yes so unfortunately due to how upstream PyTorch works you generally now need two conda envs, one for the Python side for training, and one for the C++ side in LAMMPS. The Python conda env needs full PyTorch and its associated cudatoolkit, the C++ needs to not have either of those, instead using the downloaded or built from source libtorch and generally the system CUDA. This is a pretty irritating thing to have to deal with, but just is what it is given the builds PyTorch provides.

[apoletayev]

Thank you for this - I've now tried a few permutations, and I keep getting the original error with the more recent lammps version (28Mar2023) and the latest commits of pair_allegro@stress. I sometimes get a different trace for another version of lammps (23Jun2022_update2), below.

terminate called after throwing an instance of 'std::runtime_error'
  what():  The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/nequip/nn/_graph_model.py", line 29, in forward
        pass
    model = self.model
    return (model).forward(new_data, )
            ~~~~~~~~~~~~~~ <--- HERE
  File "code/__torch__/nequip/nn/_rescale.py", line 21, in AD_logsumexp_backward
    data: Dict[str, Tensor]) -> Dict[str, Tensor]:
    model = self.model
    data0 = (model).forward(data, )
             ~~~~~~~~~~~~~~ <--- HERE
    training = self.training
    if training:
  File "code/__torch__/nequip/nn/_grad_output.py", line 71, in batch_norm
      pass
    func = self.func
    data0 = (func).forward(data, )
             ~~~~~~~~~~~~~ <--- HERE
    _17 = [torch.sum(data0["total_energy"])]
    _18 = [pos, data0["_displacement"]]
  File "code/__torch__/nequip/nn/_graph_mixin.py", line 28, in addmm
    input1 = (radial_basis).forward(input0, )
    input2 = (spharm).forward(input1, )
    input3 = (allegro).forward(input2, )
              ~~~~~~~~~~~~~~~~ <--- HERE
    input4 = (edge_eng).forward(input3, )
    input5 = (edge_eng_sum).forward(input4, )
  File "code/__torch__/allegro/nn/_allegro.py", line 113, in layer_norm
    _18 = annotate(List[Optional[Tensor]], [active_edges])
    prev_mask = torch.gt(torch.index(cutoff_coeffs, _18), 0)
    _19 = torch.nonzero(torch.gt(cutoff_coeffs, 0))
          ~~~~~~~~~~~~~ <--- HERE
    active_edges0 = torch.squeeze(_19, -1)
    _25 = torch.cat(latent_inputs_to_cat, -1)

Traceback of TorchScript, original code (most recent call last):
  File "/training/conda/environment/site-packages/nequip/nn/_graph_model.py", line 112, in forward
                new_data[k] = v
        # run the model
        data = self.model(new_data)
               ~~~~~~~~~~ <--- HERE
        return data
  File "/training/conda/environment/site-packages/nequip/nn/_rescale.py", line 144, in AD_logsumexp_backward
    def forward(self, data: AtomicDataDict.Type) -> AtomicDataDict.Type:
        data = self.model(data)
               ~~~~~~~~~~ <--- HERE
        if self.training:
            # no scaling, but still need to promote for consistent dtype behavior
  File "/training/conda/environment/site-packages/nequip/nn/_grad_output.py", line 305, in batch_norm
    
        # Call model and get gradients
        data = self.func(data)
               ~~~~~~~~~ <--- HERE
    
        grads = torch.autograd.grad(
  File "/training/conda/environment/site-packages/nequip/nn/_graph_mixin.py", line 360, in addmm
    def forward(self, input: AtomicDataDict.Type) -> AtomicDataDict.Type:
        for module in self:
            input = module(input)
                    ~~~~~~ <--- HERE
        return input
  File "/training/conda/environment/site-packages/allegro/nn/_allegro.py", line 497, in layer_norm
            cutoff_coeffs = cutoff_coeffs_all[layer_index]
            prev_mask = cutoff_coeffs[active_edges] > 0
            active_edges = (cutoff_coeffs > 0).nonzero().squeeze(-1)
                            ~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    
            # Compute latents
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

It seems you are saying that the working protocol for stress + kokkos should be (1) train with the right version of nequip 0.6.0 in a python+pytorch+cudatoolkit environment (2) deploy - with system or training environment? (3) run lammps built on libtorch + system (realistically, a loaded module) CUDA. Did I get that right / what should be the correct sequence?

[apoletayev]

Have returned to this again after giving up for a while. Getting the same errors on a variety of versions of lammps. Now building everything and training in exactly the same way as for NVT allegro except versions. If anyone has made the stress branch work I would appreciate it if they could share their workflow.

[Linux-cpp-lisp]

Hi @apoletayev ,

Hm. This continues to be very strange, we are successfully compiling this on a variety of different systems and have never seen this bug reported.

Is it always this same TorchScript trace you posted above? That would seem to not have anything to do with pair_allegro but rather with the Python side in Allegro. Your training, nequip-evaluate, etc. runs without issue?

[apoletayev]

I'm really sorry! nequip-evaluate runs without problems using the same setup as used for training:

• load pytorch with its CUDA dependencies (GPU node of a cluster)
• load virtualenv with nequip @ develop and allegro @ main (everything same as for training and deploying) which were set up to depend on the same pytorch module
• run nequip-evaluate on a trained allegro StressForceOutput model

Most recently tested LAMMPS builds (downloaded afresh every time, pair_allegro @ stress downloaded fresh every time) are 23Jun2022 updates 2 and 4, 28Mar2023 update 1, still with libtorch 1.11.0 and CUDA 11.3. The libtorch is a difference between training and lammps - but seems required for kokkos. Typical error remains as in the first post:

terminate called after throwing an instance of 'c10::Error'
  what():  expected scalar type Double but found Float
Exception raised from data_ptr<double> at aten/src/ATen/core/TensorMethods.cpp:18 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + ... (in /libtorch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + ... (in /libtorch/lib/libc10.so)
frame #2: double* at::TensorBase::data_ptr<double>() const + ... (in /libtorch/lib/libtorch_cpu.so)
frame #3: LAMMPS_NS::PairAllegroKokkos<(Precision)2>::compute(int, int) + ... (in /build/liblammps.so.0)
frame #4: LAMMPS_NS::PairHybridKokkos::compute(int, int) + ... (in /build/liblammps.so.0)

It confuses me a bit why it appears to involve libtorch_cpu.so when running on GPUs, but I have not explored how torch works inside. The no-stress builds work without complications with the same training and building (nequip @ 0.5.6, pair_allegro @ main, lammps 29Sep2021_update1/2)

[anjohan]

Right, that makes sense, since that's the function where "everything" happens.

If you run it with gdb, it should tell you the exact line where the error occurs, which will tell us what tensor is mistyped.

(And just to check: Did you try the 3264 and 6432 variants? I believe 3264 is the default.)

[apoletayev]

For lammps build patch_28Mar2023_update1 or patch_23Jun2022_update2:

• for allegro and allegro6464 and allegro3264 it is double* at::TensorBase::data_ptr<double>() const () from .../libtorch/lib/libtorch_cpu.so
• for allegro3232 and allegro6432 it is float* at::TensorBase::data_ptr<float>() const () from .../libtorch/lib/libtorch_cpu.so

I see two instances of data_ptr in PairAllegroKokkos.cpp, energies and forces, but I am not sure which one is giving the error. Sorry I'm so clumsy - I have never debugged c++. There may be more information but I get several "missing separate debuginfos" messages, and I cannot install those for not being a superuser.

Today I also trained toy models with

default_dtype: float32
model_dtype: float32

and that ran with allegro3232 on the same two lammps builds - THAT WORKS WOOO!!!
(allow_tf32 does not seem to matter)

[anjohan]

Nice to see that something is working!

If you run the gdb -ex=where -ex=r --args ... command I sent, it should give you a full stack trace with line numbers that you can copy paste here. Hopefully there's a line looking something like

#13 LAMMPS_NS::PairAllegroKokkos::compute(int, int) () at /path/to/pair_allegro_kokkos.cpp:LINENUMBER

[apoletayev]

I hear you - was looking for something like a line number but that was the best I get. Here is an example trace I get with allegro3264:

Neighbor list info ...
  update every 1 steps, delay 0 steps, check yes
  max neighbors/atom: 2000, page size: 100000
  master list distance cutoff = 7
  ghost atom cutoff = 7
  binsize = 7, bins = 4 3 2
  1 neighbor lists, perpetual/occasional/extra = 1 0 0
  (1) pair allegro3264/kk, perpetual
      attributes: full, newton on, ghost, kokkos_device
      pair build: full/bin/ghost/kk/device
      stencil: full/ghost/bin/3d
      bin: kk/device
Setting up cg/kk style minimization ...
  Unit style    : metal
  Current step  : 0
WARNING: Fixes cannot yet send exchange data in Kokkos communication, switching to classic exchange/border communication (src/KOKKOS/comm_kokkos.cpp:581)
[New Thread 0x15544d76d700 (LWP 3299750)]
terminate called after throwing an instance of 'c10::Error'
  what():  expected scalar type Double but found Float
Exception raised from data_ptr<double> at aten/src/ATen/core/TensorMethods.cpp:18 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x155554bf70eb in /path/to/libtorch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x155554bf2abe in /path/to/libtorch/lib/libc10.so)
frame #2: double* at::TensorBase::data_ptr<double>() const + 0x10b (0x1554b80b732b in /path/to/libtorch/lib/libtorch_cpu.so)
frame #3: LAMMPS_NS::PairAllegroKokkos<(Precision)2>::compute(int, int) + 0x2e78 (0x155476323678 in /path/to/lammps/build/liblammps.so.0)
frame #4: LAMMPS_NS::MinKokkos::setup(int) + 0x98a (0x155475c9187a in /path/to/lammps/build/liblammps.so.0)
frame #5: LAMMPS_NS::Minimize::command(int, char**) + 0x1ac (0x155475867bdc in /path/to/lammps/build/liblammps.so.0)
frame #6: LAMMPS_NS::Input::execute_command() + 0xbfe (0x1554757ffe7e in /path/to/lammps/build/liblammps.so.0)
frame #7: LAMMPS_NS::Input::file() + 0x166 (0x155475800276 in /path/to/lammps/build/liblammps.so.0)
frame #8: /path/to/lammps/build/lmp() [0x4015ca]
frame #9: __libc_start_main + 0xf3 (0x155474827873 in /lib64/libc.so.6)
frame #10: /path/to/lammps/build/lmp() [0x4017de]

Thread 1 "lmp" received signal SIGABRT, Aborted.
0x000015547483b8df in raise () from /lib64/libc.so.6
#0  0x000015547483b8df in raise () from /lib64/libc.so.6
#1  0x0000155474825cf5 in abort () from /lib64/libc.so.6
#2  0x0000155474c749c3 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95
#3  0x0000155474c802b6 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
#4  0x0000155474c80321 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58
#5  0x0000155474c80575 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x15555511a580 <typeinfo for c10::Error>, dest=0x155554bdefd0 <c10::Error::~Error()>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:95
#6  0x0000155554bf2aea in c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
   from /path/to/libtorch/lib/libc10.so
#7  0x00001554b80b732b in double* at::TensorBase::data_ptr<double>() const () from /path/to/libtorch/lib/libtorch_cpu.so
#8  0x0000155476323678 in LAMMPS_NS::PairAllegroKokkos<(Precision)2>::compute (this=0x9c30ffa0, eflag_in=<optimized out>, vflag_in=<optimized out>)
    at /path/to/c++/9.3.0/ext/new_allocator.h:89
#9  0x0000155475c9187a in LAMMPS_NS::MinKokkos::setup (this=0x9c30c0e0, flag=1) at /path/to/lammps/src/KOKKOS/min_kokkos.cpp:170
#10 0x0000155475867bdc in LAMMPS_NS::Minimize::command (this=0x9ceb3b70, narg=<optimized out>, arg=0x9c2fde20) at /path/to/lammps/src/minimize.cpp:57
#11 0x00001554757ffe7e in LAMMPS_NS::Input::execute_command (this=0x392bd20) at /path/to/lammps/src/input.cpp:811
#12 0x0000155475800276 in LAMMPS_NS::Input::file (this=0x392bd20) at /path/to/lammps/src/input.cpp:278
#13 0x00000000004015ca in main (argc=<optimized out>, argv=<optimized out>) at /path/to/lammps/src/main.cpp:98
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-72.el8_1.1.x86_64 hwloc-libs-1.11.9-3.el8.x86_64 libibverbs-22.3-1.el8.x86_64 libnl3-3.4.0-5.el8.x86_64 librdmacm-22.3-1.el8.x86_64 libtool-ltdl-2.4.6-25.el8.x86_64 libxcrypt-4.1.1-4.el8.x86_64 munge-libs-0.5.13-1.el8.x86_64 sssd-client-2.2.0-19.el8_1.1.x86_64

From the standpoint of looking for line numbers, they look like this. I cannot install debuginfos on the cluster, am talking to the people supporting our cluster to resolve that.

[anjohan]

Huh, that's strange. Is this with the default CMake build type, RelWithDebInfo?

You could also try to reconfigure with -DCMAKE_BUILD_TYPE=Debug in case it helps.

If nothing else works on your cluster, print statements to the rescue! When using the Kokkos version, the only two occurences of data_ptr are

  UnmanagedFloatView1D d_atomic_energy(atomic_energy_tensor.data_ptr<outputtype>(), inum);
  UnmanagedFloatView2D d_forces(forces_tensor.data_ptr<outputtype>(), ignum, 3);

If you uncomment these lines, it'll print the types of the tensors:

  //std::cout << "NequIP model output:\n";
  //std::cout << "forces:\n" << forces_tensor.cpu() << "\n";
  //std::cout << "atomic_energy:\n" << atomic_energy_tensor.cpu() << "\n";