Linking issue with static libpq compilation

[Progdrasil]

I'm currently building an app using buildozer, i've patched it to compile the version 10.12 of libpq.

Their recipe for compiling psycopg2 compiles against the static lib of libpq which works fine.
Instead it breaks at runtime with the error:

ImportError: undefined symbol: PQencryptPasswordConn

I'm running with psycopg2 2.8.4
I've read the issues #1093 and #1018 but i've confirmed that the libpq version really is 10.

[dvarrazzo]

Hello @Progdrasil

Because the problem comes your environment I don't really have a way to know what happens there. You can see that the only use of that function is guarded at compile time.

psycopg2/psycopg/psycopgmodule.c

Lines 474 to 491 in d70d3ee

	#if PG_VERSION_NUM >= 100000
	if (!conn) {
	PyErr_SetString(ProgrammingError,
	"password encryption (other than 'md5' algorithm)"
	" requires a connection or cursor");
	goto exit;
	}
	/* TODO: algo = None will block: forbid on async/green conn? */
	encrypted = PQencryptPasswordConn(conn->pgconn,
	Bytes_AS_STRING(password), Bytes_AS_STRING(user),
	algorithm != Py_None ? Bytes_AS_STRING(algorithm) : NULL);
	#else
	PyErr_SetString(NotSupportedError,
	"password encryption (other than 'md5' algorithm)"
	" requires libpq 10");
	goto exit;
	#endif

If you provide a docker image reproducing the issue in isolation we can try taking a look, thank you.

[Progdrasil]

Funny thing is I was actually getting this issue in docker yesterday fairly systematically. But today it seems to not want to reproduce...

Thats annoying. I made a repo here which should reproduce the issue as it would happen as soon as psycopg2 was imported, but it does not.
https://github.com/Progdrasil/buildozer-psycopg2-issue-1123

[grubberr]

I have similar problem
I cannot compile psycopg2 with static linking to libpq

Run inside AWS Lambda container

docker run -ti --rm --entrypoint /bin/bash --platform linux/amd64 public.ecr.aws/lambda/python:3.11

Install all needed rpm packages

yum -y install wget tar gzip gcc make flex bison perl patch

Download and compile postgresql-14.9

wget https://ftp.postgresql.org/pub/source/v14.9/postgresql-14.9.tar.gz
tar xfz postgresql-14.9.tar.gz
pushd postgresql-14.9
./configure --prefix /opt/postgresql-14.9 --without-readline --without-zlib
make
make install
popd

Download and compile psycopg2-2.9.7 with static linking to libpq

wget https://files.pythonhosted.org/packages/f7/fa/6e6bb7a7bbe4e02b35aa2fc009fb53221663a5e07e333b72cb5a85e4dbb0/psycopg2-2.9.7.tar.gz
tar xfz psycopg2-2.9.7.tar.gz
pushd psycopg2-2.9.7
sed -i 's@pg_config =.*@pg_config = /opt/postgresql-14.9/bin/pg_config@' setup.cfg
sed -i 's@static_libpq =.*@static_libpq = 1@' setup.cfg
python3 setup.py build
python3 setup.py install
popd

Import compile psycopg2 module

bash-4.2# python3 -m psycopg2
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "/var/lang/lib/python3.11/site-packages/psycopg2-2.9.7-py3.11-linux-x86_64.egg/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: /var/lang/lib/python3.11/site-packages/psycopg2-2.9.7-py3.11-linux-x86_64.egg/psycopg2/_psycopg.cpython-311-x86_64-linux-gnu.so: undefined symbol: scram_SaltedPassword

Any idea how can we eliminate this problem ?

[minhio]

I have similar problem I cannot compile psycopg2 with static linking to libpq

Run inside AWS Lambda container

docker run -ti --rm --entrypoint /bin/bash --platform linux/amd64 public.ecr.aws/lambda/python:3.11

Install all needed rpm packages

yum -y install wget tar gzip gcc make flex bison perl patch

Download and compile postgresql-14.9

wget https://ftp.postgresql.org/pub/source/v14.9/postgresql-14.9.tar.gz
tar xfz postgresql-14.9.tar.gz
pushd postgresql-14.9
./configure --prefix /opt/postgresql-14.9 --without-readline --without-zlib
make
make install
popd

Download and compile psycopg2-2.9.7 with static linking to libpq

wget https://files.pythonhosted.org/packages/f7/fa/6e6bb7a7bbe4e02b35aa2fc009fb53221663a5e07e333b72cb5a85e4dbb0/psycopg2-2.9.7.tar.gz
tar xfz psycopg2-2.9.7.tar.gz
pushd psycopg2-2.9.7
sed -i 's@pg_config =.*@pg_config = /opt/postgresql-14.9/bin/pg_config@' setup.cfg
sed -i 's@static_libpq =.*@static_libpq = 1@' setup.cfg
python3 setup.py build
python3 setup.py install
popd

Import compile psycopg2 module

bash-4.2# python3 -m psycopg2
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "/var/lang/lib/python3.11/site-packages/psycopg2-2.9.7-py3.11-linux-x86_64.egg/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: /var/lang/lib/python3.11/site-packages/psycopg2-2.9.7-py3.11-linux-x86_64.egg/psycopg2/_psycopg.cpython-311-x86_64-linux-gnu.so: undefined symbol: scram_SaltedPassword

Any idea how can we eliminate this problem ?

@grubberr were you able to solve this?

[grubberr]

@minhio no, we did some work-around :(

[jarias1]

I have similar problem I cannot compile psycopg2 with static linking to libpq

Run inside AWS Lambda container

docker run -ti --rm --entrypoint /bin/bash --platform linux/amd64 public.ecr.aws/lambda/python:3.11

Install all needed rpm packages

yum -y install wget tar gzip gcc make flex bison perl patch

Download and compile postgresql-14.9

wget https://ftp.postgresql.org/pub/source/v14.9/postgresql-14.9.tar.gz
tar xfz postgresql-14.9.tar.gz
pushd postgresql-14.9
./configure --prefix /opt/postgresql-14.9 --without-readline --without-zlib
make
make install
popd

Download and compile psycopg2-2.9.7 with static linking to libpq

wget https://files.pythonhosted.org/packages/f7/fa/6e6bb7a7bbe4e02b35aa2fc009fb53221663a5e07e333b72cb5a85e4dbb0/psycopg2-2.9.7.tar.gz
tar xfz psycopg2-2.9.7.tar.gz
pushd psycopg2-2.9.7
sed -i 's@pg_config =.*@pg_config = /opt/postgresql-14.9/bin/pg_config@' setup.cfg
sed -i 's@static_libpq =.*@static_libpq = 1@' setup.cfg
python3 setup.py build
python3 setup.py install
popd

Import compile psycopg2 module

bash-4.2# python3 -m psycopg2
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "/var/lang/lib/python3.11/site-packages/psycopg2-2.9.7-py3.11-linux-x86_64.egg/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: /var/lang/lib/python3.11/site-packages/psycopg2-2.9.7-py3.11-linux-x86_64.egg/psycopg2/_psycopg.cpython-311-x86_64-linux-gnu.so: undefined symbol: scram_SaltedPassword

Any idea how can we eliminate this problem ?

same issue here, I compiled it with option static_libpq=1 but receive error that says: undefined symbol: pg_tolower