AWS Lambda with psycopg2 (compiled with PostgreSQL 13.4)

[billglennon]

I am basically trying to do the following for AWS Lambda for psycopg2.
https://github.com/jkehler/awslambda-psycopg2

I am trying with the following source code bases and I get the error below.
Python 3.8.1
Psycopg2 2.9.3
PostgreSQL 13.4

psycopg2 setup.cfg config file

psycopg2-2.9.3/setup.cfg
pg_config =
static_libpq = 1
libraries = ssl crypto

However, when I try with PostgreSQL 11.14 it works.
Python 3.8.1
Psycopg2 2.9.3
PostgreSQL 11.14

I also tried with PostgreSQL 12.9 and I get the same error.

Of the web sites I found with this process, all the database versions were PostgreSQL 11 or lower.

Did something change in 12 and above to cause this issue with psycopg2?

Any help/pointers would be appreciated.

Error Message:

>>> import psycopg2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/psycopg2-2.9.3-py3.8-linux-x86_64.egg/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: /usr/local/lib/python3.8/site-packages/psycopg2-2.9.3-py3.8-linux-x86_64.egg/psycopg2/_psycopg.cpython-38-x86_64-linux-gnu.so: undefined symbol: pg_tolower

[dvarrazzo]

I am basically trying to do the following for AWS Lambda for psycopg2.
https://github.com/jkehler/awslambda-psycopg2

This is not a project we maintain. The author never got in touch with us. I have no interest in investigating into it.

[billglennon]

OK. No problem. I was thinking it was something between psycopg2 and new versions of PostgreSQL with static_libpq in the setup.cfg for psycopg2. FYI - If change static_libpq =0 then it works with PostgereSQL 13.4. This is has nothing to do with Lambda at this point. This is compiling the 3 things from source (Python, psycopg2, and PostgreSQL). I get this error on the local server I am compiling it and running Python on.

So to recap (nothing running on Lambda) - all local on a server:
Does not work
Does not work
Python 3.8.11
Psycopg2 2.9.3
PostgreSQL 13.4

psycopg2-2.9.3/setup.cfg
pg_config = <pg_config location>
static_libpq = 1
libraries = ssl crypto

Does work
Python 3.8.11
Psycopg2 2.9.3
PostgreSQL 13.4

psycopg2-2.9.3/setup.cfg
pg_config = <pg_config location>
static_libpq = 0
libraries = ssl crypto

Does work
Python 3.8.11
Psycopg2 2.9.3
PostgreSQL 11.14

psycopg2-2.9.3/setup.cfg
pg_config = <pg_config location>
static_libpq = 1
libraries = ssl crypto

[szymic1]

Do you know where I can find compiled psycopg2 for Lambda? You mentioned about https://github.com/jkehler/awslambda-psycopg2 but it use very old libraries (from 2019, Postgres 9.6). I've tried to compiled using Postgres 13.5 + Psycopg2 2.9.3 but I'm failed. I've got error:
x86_64-linux-gnu-gcc: error: /home/michal/Documents/psycopg/postgresql-13.5/src/lib/libpq.a: No such file or directory

[szymic1]

BTW in another post @dvarrazzo wrote "The alternative of using psycopg2-binary is to install the libpq. If you choose to run your software on a platform you don't have control of it's not really a problem we can solve.

However the current binary packages, shipping OpenSSL 1.1.x, have proven stable enough and not as problematic as the versions used when that disclaimer warning was written. I think you can use it no problem."

[szymic1]

But wat can be interesting is compile psycopg2 for ARM (Graviton) platform, from September Lambda can be run on ARM (what is cheaper)! @billglennon if you compile psycopg2 on ARM let me know :)