refactor: upgrade to psycopg3[binary]==3.3.3

[bhardwajparth51]

Description

Part of #10258 upgrade.

refactor

This PR upgrades Open Library's underlying PostgreSQL driver from psycopg2
to psycopg[binary]==3.3.3. Following the successful migration of the Open
Library database from PostgreSQL 9.3 to 18, this update ensures our application
bindings are modernized and can leverage psycopg3 natively for better
performance and future async execution.

Changes

1. Dependencies: Updated requirements.txt to use psycopg[binary]==3.3.3.
2. Config: Added driver: psycopg explicitly to db_parameters in
   conf/coverstore.yml, conf/infobase.yml, and conf/openlibrary.yml.
3. Namespaces: Refactored all module exception imports to use psycopg.errors
   across core/db.py, core/edits.py, core/imports.py, and scripts/migrations/.
4. Solr Builder: Upgraded driver instantiations and typings natively in
   solr_builder.py. Removed cur.itersize which is unsupported in psycopg3.
5. Config Patch (openlibrary/config.py): Added a temporary compatibility
   shim for infogami's parse_db_parameters(), which currently strips the
   driver key. The patch preserves it until the submodule is updated natively.
   Marked with # TODO

Known Limitation

infogami is a submodule that still internally references psycopg2. The config
patch in openlibrary/config.py is a deliberate stopgap —to be tracked in #10258.

Future Proofing for FastAPI Async

Since psycopg3 natively supports asyncio, this sets up future FastAPI
endpoints to use AsyncConnectionPool without additional driver changes.

Testing

1. Run docker compose build home to pull in the new dependencies.
2. Run docker compose run --rm home make test-py successfully against the
   revised stack (3008 passed, 0 failures).
3. Boot up the local webserver (docker compose up) and verify that standard
   database components resolve properly.
4. Confirmed driver: psycopg is present in web.config.db_parameters
   after config load.

Stakeholders

@mekarpeles @RayBB

[bhardwajparth51]

vendor/infogami/requirements_test.txt still has psycopg2. I've left vendored test requirements untouched for now as part of the transitional phase, but I've added shims in config.py to ensure OL remains fully compatible with psycopg3 throughout

[RayBB]

Maybe the approach we should take here is to use it in our application in places that we currently are using it directly, and then do a separate pull request for Infogami. Because I don't think that we are going to be able to just patch in Kogami as a long-term solution. We need to fully improve it at its source.

[bhardwajparth51]

Agreed, patching config.py is definitely a code smell. We should fix it at the source in the infogami repo instead.

Just to make sure we're on the same page for this PR: I am thinking to do a hybrid setup where we have both psycopg2 and psycopg[binary]==3.3.3 in requirements.txt. That way infogami can keep using v2 for now, while we upgrade direct usages like solr_builder to v3.

Or should we first, fix the internetarchive/infogami first to update its driver parsing, and then we do a single, clean switch later?"

[RayBB]

Let's first the second one first to do a clean switch. I'm also not as familiar with solr builder so it's better to start with the part were more familiar with. That being said, I think there may be a case to try just using psycopg3 directly without web.py in our main codebase to see what the DX is. I'm not sure there's a specific need for us to be using the connection through web.py. What do you think?

…

On Mon, Mar 30, 2026, 1:23 PM Parth Bhardwaj ***@***.***> wrote: *bhardwajparth51* left a comment (internetarchive/openlibrary#12212) <#12212?email_source=notifications&email_token=AAHA5AIUB7FN7QGG77A2FMD4TLJVJA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJVG44TINZQGQ22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJNLQOJPWG33NNVSW45C7N5YGK3S7MNWGSY3L#issuecomment-4157947045> Agreed, patching config.py is definitely a code smell. We should fix it at the source in the infogami repo instead. Just to make sure we're on the same page for this PR: I am thinking to do a hybrid setup where we have both psycopg2 and psycopg[binary]==3.3.3 in requirements.txt. That way infogami can keep using v2 for now, while we upgrade direct usages like solr_builder to v3. Or should we first, fix the internetarchive/infogami first to update its driver parsing, and then we do a single, clean switch later?" — Reply to this email directly, view it on GitHub <#12212?email_source=notifications&email_token=AAHA5AIUB7FN7QGG77A2FMD4TLJVJA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJVG44TINZQGQ22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJNLQOJPWG33NNVSW45C7N5YGK3S7MNWGSY3L#issuecomment-4157947045>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHA5AOTKNQYZRTCJQXT6CD4TLJVJAVCNFSM6AAAAACXC65HSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCNJXHE2DOMBUGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>