long-running commands and PostgreSQL

[silenius]

We've the following issue when import_observations (in production) it looks like Django doesn't reconnect automatically when the connection is dropped (?), maybe related to https://code.djangoproject.com/ticket/34914:

(gbif-alert) www2-py311-2024Q3% DJANGO_SETTINGS_MODULE="djangoproject.settings.production" python manage.py import_observations
Fri Jan 24 12:04:04 2025: (Re)importing all observations
Fri Jan 24 12:04:04 2025: No DWCA file provided, we'll generate and get a new GBIF download
Fri Jan 24 12:04:04 2025: Triggering a GBIF download and waiting for it - this can be long...
Fri Jan 24 13:10:07 2025: Observations downloaded
Fri Jan 24 13:10:07 2025: We now have a (locally accessible) source dwca, real import is starting. We'll use a transaction and put the website in maintenance mode
Traceback (most recent call last):
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.OperationalError: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/www/sites/gbif-alert/manage.py", line 22, in <module>
    main()
  File "/usr/local/www/sites/gbif-alert/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/www/sites/gbif-alert/dashboard/management/commands/import_observations.py", line 311, in handle
    current_data_import = DataImport.objects.create(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/models/query.py", line 658, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/models/base.py", line 877, in save_base
    updated = self._save_table(
              ^^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/models/base.py", line 1020, in _save_table
    results = self._do_insert(
              ^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/models/base.py", line 1061, in _do_insert
    return manager._insert(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/models/query.py", line 1805, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/venvs/gbif-alert/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.OperationalError: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

cc @sronveaux

[silenius]

So it works if the DATABASES is configured to connect directly to the PostgreSQL server:

DATABASES = {
    'default': {
        'ENGINE': "django.contrib.gis.db.backends.postgis",
        'NAME': "gbif-alert",
        'USER': "some_user",
        'PASSWORD': "some_password",
        'HOST': "pg15.prod.lan",
        'PORT': "5432",
    }
}

But in our configuration we have an HAProxy and Pgbouncer(s) in front of our PostgreSQL instance(s) (with a timeout of 30min), so the connection goes to HAProxy -> Pgbouncer -> PostgreSQL and as the import takes > 30 min the connection is closed. I've made a workaround but it would be good is this could be fixed :) maybe it's just a call to close_old_connections() at some step in handle() .. to investigate 😄