Protobuf problem when building Lammps after TF and DeepMD-kit

[Mskater]

Hi All,
I have installed and can verify Tensorflow 2.0.0, and DeepMD-kit with the python interface now brings up the help screen with "dp -h".
But when I try to "make mpi -j4" for Lammps, I get errors saying tensorflow files were made with old Protobuf (2.6.1 original install on Ubuntu 16.04), and please regenerate with newer version, and things are marked 'final but are not virtual'.
So my question is, do I have to reinstall/rebuild tensorflow with a newer Protobuf (what version for Protobuf and protoc?) Or if I delete my now newer Protobuf 3.13, maybe "make" won't complain when building LAMMPs? Which direction to go? Or something else?

Thanks for any help.

Environment variables:
marie@marie-Precision-7530:~/build/lammpsworkspace/lammps-3Mar20/src$ env
XDG_VTNR=7
XDG_SESSION_ID=c12
CLUTTER_IM_MODULE=xim
XDG_GREETER_DATA_DIR=/var/lib/lightdm-data/marie
SESSION=ubuntu
GPG_AGENT_INFO=/home/marie/.gnupg/S.gpg-agent:0:1
TERM=xterm-256color
SHELL=/bin/bash
VTE_VERSION=4205
CONDA_SHLVL=0
QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1
WINDOWID=67108874
UPSTART_SESSION=unix:abstract=/com/ubuntu/upstart-session/1001/2041
GNOME_KEYRING_CONTROL=
GTK_MODULES=gail:atk-bridge:unity-gtk-module
USER=marie
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.oga=00;36:.opus=00;36:.spx=00;36:.xspf=00;36:
QT_ACCESSIBILITY=1
CONDA_EXE=/home/marie/miniconda3/bin/conda
XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0
XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
SSH_AUTH_SOCK=/run/user/1001/keyring/ssh
DESKTOP_MODE=1
DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
_CE_CONDA=
XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/usr/share/upstart/xdg:/etc/xdg
PATH=/home/marie/miniconda3/condabin:/home/marie/bin:/home/marie/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/marie/bin::/home/usr/bin::/bin::/.local/bin::/home/marie/tensorflow_root::/home/marie/tensorflow_root/include/tensorflow/core::/home/marie/deepmd_root::/home/marie/miniconda3/bin::/home/marie/miniconda3/condabin::/home/marie/.local/bin::/usr/local/sbin::/usr/local/bin::/usr/sbin::/usr/bin::/sbin::/snap/bin:
DESKTOP_SESSION=ubuntu
QT_IM_MODULE=ibus
QT_QPA_PLATFORMTHEME=appmenu-qt5
XDG_SESSION_TYPE=x11
JOB=unity-settings-daemon
PWD=/home/marie/build/lammpsworkspace/lammps-3Mar20/src
XMODIFIERS=@im=ibus
GNOME_KEYRING_PID=
LANG=en_CA.UTF-8
GDM_LANG=en_CA
MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
IM_CONFIG_PHASE=1
COMPIZ_CONFIG_PROFILE=ubuntu
GDMSESSION=ubuntu
_CE_M=
SESSIONTYPE=gnome-session
GTK2_MODULES=overlay-scrollbar
HOME=/home/marie
XDG_SEAT=seat0
SHLVL=1
LANGUAGE=en_CA:en
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
UPSTART_INSTANCE=
CONDA_PYTHON_EXE=/home/marie/miniconda3/bin/python
UPSTART_EVENTS=xsession started
XDG_SESSION_DESKTOP=ubuntu
LOGNAME=marie
COMPIZ_BIN_PATH=/usr/bin/
QT4_IM_MODULE=xim
XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share:/usr/share:/var/lib/snapd/desktop:/var/lib/snapd/desktop
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-0MPkkiJjmR
LESSOPEN=| /usr/bin/lesspipe %s
INSTANCE=
UPSTART_JOB=unity7
XDG_RUNTIME_DIR=/run/user/1001
DISPLAY=:0
XDG_CURRENT_DESKTOP=Unity
GTK_IM_MODULE=ibus
LESSCLOSE=/usr/bin/lesspipe %s %s
XAUTHORITY=/home/marie/.Xauthority
_=/usr/bin/env
OLDPWD=/home/marie/build/lammpsworkspace/lammps-3Mar20

[njzjz]

How do you build C++ version of the TensorFlow? There are protobuf headers in the build directory, and you should use them.

[Mskater]

Thanks but I don't know anything about how to do that.
Can you give some further direction?
Thanks

[Mskater]

Does that also mean I must build tensorflow over again?

[njzjz]

When you build tensorflow, it downloads a specific version of protobuf:
https://github.com/tensorflow/tensorflow/blob/64c3d382cadf7bbe8e7e99884bede8284ff67f56/tensorflow/workspace.bzl#L420-L442

You need to copy headers from the build directory

rsync -avzh --include '*/' --include '*.h' --include '*.inc' --exclude '*' bazel-tensorflow/external/com_google_protobuf/src/google/ $tensorflow_root/include/google/

[njzjz]

You can also download https://github.com/protocolbuffers/protobuf/archive/310ba5ee72661c081129eb878c1bbcec936b20f0.tar.gz and copy headers (*.h and *.inc) from it.

[Mskater]

Thanks. I now did copy over all the protobuf headers, they were downloaded into a different directory than the instructions say. But I still get the same errors. I already went thru rebuilding Tensorflow 2.1.0 too, before copying headers. Please see the following errors. Can you tell me exactly what to do to "regenerate the files with a newer version of protoc" and get rid of the other errors? Is there some way to copy over the new tensorflow files into deepmd_root/include?

I also upgraded to Ubuntu 18.04 before the above actions, thinking it would solve something.

ERROR OUTPUT
marie@marie-Precision-7530:~/build/lammpsworkspace/lammps-3Mar20/src$ make mpi -j4
Gathering installed package information (may take a little while)
make[1]: Entering directory '/home/marie/build/lammpsworkspace/lammps-3Mar20/src'
make[1]: 'lmpinstalledpkgs.h' is up to date.
Gathering git version information
make[1]: Leaving directory '/home/marie/build/lammpsworkspace/lammps-3Mar20/src'
Compiling LAMMPS for machine mpi
make[1]: Entering directory '/home/marie/build/lammpsworkspace/lammps-3Mar20/src/Obj_mpi'
make[1]: Leaving directory '/home/marie/build/lammpsworkspace/lammps-3Mar20/src/Obj_mpi'
make[1]: Entering directory '/home/marie/build/lammpsworkspace/lammps-3Mar20/src/Obj_mpi'
mpicxx -g -O3 -DLAMMPS_GZIP -DLAMMPS_MEMALIGN=64 -DLMP_PYTHON -std=c++11 -DHIGH_PREC -I/home/marie/deepmd_root/include -I/home/marie/deepmd_root/include -I/home/marie/deepmd_root/include/deepmd -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1 -I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7 -c ../force.cpp
mpicxx -g -O3 -DLAMMPS_GZIP -DLAMMPS_MEMALIGN=64 -DLMP_PYTHON -std=c++11 -DHIGH_PREC -I/home/marie/deepmd_root/include -I/home/marie/deepmd_root/include -I/home/marie/deepmd_root/include/deepmd -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1 -I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7 -c ../modify.cpp
mpicxx -g -O3 -DLAMMPS_GZIP -DLAMMPS_MEMALIGN=64 -DLMP_PYTHON -std=c++11 -DHIGH_PREC -I/home/marie/deepmd_root/include -I/home/marie/deepmd_root/include -I/home/marie/deepmd_root/include/deepmd -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1 -I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7 -c ../fix_dplr.cpp
mpicxx -g -O3 -DLAMMPS_GZIP -DLAMMPS_MEMALIGN=64 -DLMP_PYTHON -std=c++11 -DHIGH_PREC -I/home/marie/deepmd_root/include -I/home/marie/deepmd_root/include -I/home/marie/deepmd_root/include/deepmd -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1 -I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7 -c ../pair_nnp.cpp
In file included from /home/marie/deepmd_root/include/tensorflow/core/public/session.h:22:0,
from /home/marie/deepmd_root/include/deepmd/common.h:3,
from /home/marie/deepmd_root/include/deepmd/NNPInter.h:3,
from ../pair_nnp.h:24,
from ../style_pair.h:109,
from ../force.cpp:22:
/home/marie/deepmd_root/include/tensorflow/core/framework/device_attributes.pb.h:17:2: error: #error This file was generated by an older version of protoc which is
#error This file was generated by an older version of protoc which is
^~~~~
/home/marie/deepmd_root/include/tensorflow/core/framework/device_attributes.pb.h:18:2: error: #error incompatible with your Protocol Buffer headers. Please
#error incompatible with your Protocol Buffer headers. Please
^~~~~
/home/marie/deepmd_root/include/tensorflow/core/framework/device_attributes.pb.h:19:2: error: #error regenerate this file with a newer version of protoc.
#error regenerate this file with a newer version of protoc.
^~~~~
In file included from /home/marie/deepmd_root/include/tensorflow/core/public/session.h:22:0,
from /home/marie/deepmd_root/include/deepmd/common.h:3,
from /home/marie/deepmd_root/include/deepmd/NNPInter.h:3,
from ../pair_nnp.h:24,
from ../fix_dplr.h:12,
from ../style_fix.h:22,
from ../modify.cpp:17:
/home/marie/deepmd_root/include/tensorflow/core/framework/device_attributes.pb.h:17:2: error: #error This file was generated by an older version of protoc which is
#error This file was generated by an older version of protoc which is
^~~~~
/home/marie/deepmd_root/include/tensorflow/core/framework/device_attributes.pb.h:18:2: error: #error incompatible with your Protocol Buffer headers. Please
#error incompatible with your Protocol Buffer headers. Please
^~~~~
/home/marie/deepmd_root/include/tensorflow/core/framework/device_attributes.pb.h:19:2: error: #error regenerate this file with a newer version of protoc.

AND these types of errors:

/home/marie/deepmd_root/include/tensorflow/core/protobuf/config.pb.h:2166:35: error: ‘google::protobuf::uint8* tensorflow::ConfigProto::InternalSerializeWithCachedSizesToArray(google::protobuf::uint8*) const’ marked ‘final’, but is not virtual
::PROTOBUF_NAMESPACE_ID::uint8* InternalSerializeWithCachedSizesToArray(
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/marie/deepmd_root/include/tensorflow/core/protobuf/config.pb.h:3116:38: error: ‘InternalMetadataWithArena’ in namespace ‘google::protobuf::internal’ does not name a type
::PROTOBUF_NAMESPACE_ID::internal::InternalMetadataWithArena internal_metadata;

[njzjz]

You can modify Makefile.package in your lammps directory to assign your include directory.

[Mskater]

Thanks! I now did transfer over new tensorflow core files made with a newer protobuf (so that the versions match) to deepmd_root/include directory. That fixed this particular problem.