libphobos_admin.so: undefined symbol: layout_reconstruct

[cardenas-in2p3]

Dear all,

We would like to evaluate phobos to run in our heterogeneous infrastructure.

I search to compile and run phobos 3.2.0 on Red Hat Enterprise Linux like and Ubuntu.

Compiling and running phobos 3.2.0 on Rocky Linux 8.10 is OK. In Ubuntu 22.04 it compiles fine but I have "undefined symbol" error in runtime, something was wrong in link time:
phobos-master$ source tests/test_env.sh
phobos-master$ ./scripts/phobos_db setup_db -s -d phobos_test -p phobos

Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
import(pkg_name)
File "/home/user/phobos-master/src/cli/build/lib.linux-x86_64-3.10/phobos/init.py", line 33, in
from phobos.cli.common import PhobosActionContext
File "/home/user/phobos-master/src/cli/build/lib.linux-x86_64-3.10/phobos/cli/common/init.py", line 40, in
from phobos.core import cfg
File "/home/user/phobos-master/src/cli/build/lib.linux-x86_64-3.10/phobos/core/cfg.py", line 30, in
from phobos.core.ffi import (LIBPHOBOS, ResourceFamily)
File "/home/user/phobos-master/src/cli/build/lib.linux-x86_64-3.10/phobos/core/ffi.py", line 52, in
LIBPHOBOS_ADMIN = CDLL(LIBPHOBOS_ADMIN_NAME)
File "/usr/lib/python3.10/ctypes/init.py", line 374, in init
self._handle = _dlopen(self._name, mode)
OSError: /home/user/phobos-master/src/admin/.libs/libphobos_admin.so: undefined symbol: layout_reconstruct

The "layout_reconstruct" is referenced on libphobos_store.so and libphobos_admin.so:
phobos-master$ nm src/admin/.libs/libphobos_admin.so | grep layout_reconstruct

U layout_reconstruct

phobos-master$ nm src/store/.libs/libphobos_store.so | grep layout_reconstruct

000000000000c560 T layout_reconstruct

Both libraries are loaded by ffi.py libphobos_store.so before that libphobos_admin.so:

phobos-master$ cat -n src/cli/build/lib.linux-x86_64-3.10/phobos/core/ffi.py

44	LIBPHOBOS_NAME = "libphobos_store.so"
45	LIBPHOBOS = CDLL(LIBPHOBOS_NAME)
46	# FIXME we should call phobos_fini(). But this can only be done once we don't
47	# need phobos anymore. Since phobos_fini() only frees memory currently, this is
48	# not an issue.
49	LIBPHOBOS.phobos_init()
50	
51	LIBPHOBOS_ADMIN_NAME = "libphobos_admin.so"
52	LIBPHOBOS_ADMIN = CDLL(LIBPHOBOS_ADMIN_NAME)
53

The system uses:

libglib2.0-0 and libglib2.0-dev version 2.72 for GLib library of C routines
python3.10 and libpython3.10

Any ideas?

[cardenas-in2p3]

pg_config shows:

phobos-master$ pg_config

BINDIR = /usr/lib/postgresql/14/bin
DOCDIR = /usr/share/doc/postgresql-doc-14
HTMLDIR = /usr/share/doc/postgresql-doc-14
INCLUDEDIR = /usr/include/postgresql
PKGINCLUDEDIR = /usr/include/postgresql
INCLUDEDIR-SERVER = /usr/include/postgresql/14/server
LIBDIR = /usr/lib/x86_64-linux-gnu
PKGLIBDIR = /usr/lib/postgresql/14/lib
LOCALEDIR = /usr/share/locale
MANDIR = /usr/share/postgresql/14/man
SHAREDIR = /usr/share/postgresql/14
SYSCONFDIR = /etc/postgresql-common
PGXS = /usr/lib/postgresql/14/lib/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--runstatedir=/run' '--disable-maintainer-mode' '--disable-dependency-tracking' '--with-tcl' '--with-perl' '--with-python' '--with-pam' '--with-openssl' '--with-libxml' '--with-libxslt' '--mandir=/usr/share/postgresql/14/man' '--docdir=/usr/share/doc/postgresql-doc-14' '--sysconfdir=/etc/postgresql-common' '--datarootdir=/usr/share/' '--datadir=/usr/share/postgresql/14' '--bindir=/usr/lib/postgresql/14/bin' '--libdir=/usr/lib/x86_64-linux-gnu/' '--libexecdir=/usr/lib/postgresql/' '--includedir=/usr/include/postgresql/' '--with-extra-version= (Ubuntu 14.19-0ubuntu0.22.04.1)' '--enable-nls' '--enable-thread-safety' '--enable-debug' '--enable-dtrace' '--disable-rpath' '--with-uuid=e2fs' '--with-gnu-ld' '--with-gssapi' '--with-ldap' '--with-pgport=5432' '--with-system-tzdata=/usr/share/zoneinfo' 'AWK=mawk' 'MKDIR_P=/bin/mkdir -p' 'PROVE=/usr/bin/prove' 'PYTHON=/usr/bin/python3' 'TAR=/bin/tar' 'XSLTPROC=xsltproc --nonet' 'CFLAGS=-g -O2 -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer' 'LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now' '--enable-tap-tests' '--with-icu' '--with-llvm' 'LLVM_CONFIG=/usr/bin/llvm-config-14' 'CLANG=/usr/bin/clang-14' '--with-lz4' '--with-systemd' '--with-selinux' 'build_alias=x86_64-linux-gnu' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security'
CC = gcc
CPPFLAGS = -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -g -O2 -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer
CFLAGS_SL = -fPIC
LDFLAGS = -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -Wl,--as-needed
LDFLAGS_EX =
LDFLAGS_SL =
LIBS = -lpgcommon -lpgport -lselinux -llz4 -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lz -lreadline -lm
VERSION = PostgreSQL 14.19 (Ubuntu 14.19-0ubuntu0.22.04.1)

[patlucas]

Hi @cardenas-in2p3 ,

Thanks a lot for opening this issue.

We apologize for the delayed response but this cea-hpc/phobos repository is only maintained for backward compatibility. It is no longer the active repository. For active issues and pull requests, we migrated to https://github.com/phobos-storage/phobos .

It's true that we are more familiar to the Redhat/CentOS/AlmaLinux/Rocky RPM environment that the Debian/Ubuntu ".deb". The Phobos continuous integration framework runs on the "Redhat" family and Phobos project currently proposes an RPM packaging.

But, it should work on Ubuntu too. We start to look for understanding your problem and come back as soon as possible.

[patlucas]

We saw many unexpected environment differences between all the Redhat/CentOS/AlmaLinux that we test and your ubuntu:22.04 .

We are currently working on the integration of all the corresponding patches into the master branch of Phobos.

Among the differences, there is the compiler/linker behaviour. In order to resolve the layout_reconstruct function, we need to explicitly add on ubuntu 22.04 the libpho_layout.la dependency to the libphobos_admin_la_LIBADD src/adminMakefile.am target .

We will come back to you as soon as the corresponding commits are integrated to master.

[cardenas-in2p3]

Hi @patlucas

Thank you for your response.

The tests with Ubuntu are not relevant; we do not use Ubuntu in production, so it is not worth working on.

Our production platform Red Hat Enterprise Linux and phobos works well on Red Hat, so we will continue our testing on this platform.

Regards,