NetBox v4.0.7 start fails with: "connection failed: connection to server at <ip>, port 5432 failed: session is read-only"

[ngotzmann]

I am trying to install NetBox 4.0.7 via the netbox community helm chart.
And I am using a postgres database with pgpool, I am getting following error on the netbox startup:

django.db.utils.OperationalError: connection failed: connection to server at "ip", port 5432 failed: session is read-only
The full stack trace is at the end of this post.

I found following discussion, which indicates a similiar problem but with an older netbox version and there is also not really a solution: #13994

I applied set the variable DISABLE_SERVER_SIDE_CURSORS/externalDatabase.disableServerSideCursors to true and the externalDatabase.connMaxAge to 300, but the problem still occours.

I can see that the migration was successful, but the connection after the migration is just not working.
Does someone encountered the same or an similiar issue and know how to fix it?

Fullstack trace:

django.db.utils.OperationalError: connection failed: connection to server at "ip", port 5432 failed: session is read-only                                                                                                          
[ Use DB_WAIT_DEBUG=1 in netbox.env to print full traceback for errors here ]                                                                                                                                                                
⏳ Waiting on DB... (0s / 30s)                                                                                                                                                                                                               
django.db.utils.OperationalError: connection failed: connection to server at "ip", port 5432 failed: session is read-only                                                                                                          
[ Use DB_WAIT_DEBUG=1 in netbox.env to print full traceback for errors here ]                                                                                                                                                                
⏳ Waiting on DB... (3s / 30s)                                                                                                                                                                                                               
django.db.utils.OperationalError: connection failed: connection to server at "ip", port 5432 failed: session is read-only                                                                                                          
[ Use DB_WAIT_DEBUG=1 in netbox.env to print full traceback for errors here ]                                                                                                                                                                
⏳ Waiting on DB... (6s / 30s)                                                                                                                                                                                                               
django.db.utils.OperationalError: connection failed: connection to server at "ip", port 5432 failed: session is read-only                                                                                                          
[ Use DB_WAIT_DEBUG=1 in netbox.env to print full traceback for errors here ]                                                                                                                                                                
⏳ Waiting on DB... (9s / 30s)                                                                                                                                                                                                               
⚙️ Applying database migrations                                                                                                                                                                                                               
Database is not ready                                                                                                                                                                                                                        
🧬 loaded config '/etc/netbox/config/configuration.py'                                                                                                                                                                                       
🧬 loaded config '/etc/netbox/config/extra.py'                                                                                                                                                                                               
🧬 loaded config '/etc/netbox/config/logging.py'                                                                                                                                                                                             
🧬 loaded config '/etc/netbox/config/plugins.py'                                                                                                                                                                                             
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/connection.py", line 119, in connect
    raise last_ex.with_traceback(None)
psycopg.OperationalError: connection failed: connection to server at "ip", port 5432 failed: session is read-only

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/netbox/netbox/./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 100, in handle
    self.check(databases=[database])
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/base.py", line 486, in check
    all_issues = checks.run_checks(
                 ^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/checks/registry.py", line 88, in run_checks
    new_errors = check(app_configs=app_configs, databases=databases)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/checks/urls.py", line 14, in check_url_config
    return check_resolver(resolver)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/checks/urls.py", line 24, in check_resolver
    return check_method()
           ^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/urls/resolvers.py", line 519, in check
    for pattern in self.url_patterns:
                   ^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/utils/functional.py", line 47, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/urls/resolvers.py", line 738, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
                       ^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/utils/functional.py", line 47, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/urls/resolvers.py", line 731, in urlconf_module
    return import_module(self.urlconf_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/netbox/netbox/netbox/urls.py", line 27, in <module>
    path('circuits/', include('circuits.urls')),
                      ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/urls/conf.py", line 39, in include
    urlconf_module = import_module(urlconf_module)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/netbox/netbox/circuits/urls.py", line 4, in <module>
    from . import views
  File "/opt/netbox/netbox/circuits/views.py", line 5, in <module>
    from dcim.views import PathTraceView
  File "/opt/netbox/netbox/dcim/views.py", line 19, in <module>
    from extras.views import ObjectConfigContextView
  File "/opt/netbox/netbox/extras/views.py", line 690, in <module>
    class ObjectChangeListView(generic.ObjectListView):
  File "/opt/netbox/netbox/extras/views.py", line 691, in ObjectChangeListView
    queryset = ObjectChange.objects.valid_models()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/netbox/extras/querysets.py", line 159, in valid_models
    content_types = ContentType.objects.get_for_models(*apps.get_models()).values()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/contrib/contenttypes/models.py", line 91, in get_for_models
    for ct in cts:
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/query.py", line 400, in __iter__
    self._fetch_all()
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/query.py", line 1928, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1560, in execute_sql
    cursor = self.connection.cursor()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 316, in cursor
    return self._cursor()
           ^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 292, in _cursor
    self.ensure_connection()
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 274, in ensure_connection
    with self.wrap_database_errors:
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection
    self.connect()
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 256, in connect
    self.connection = self.get_new_connection(conn_params)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 277, in get_new_connection
    connection = self.Database.connect(**conn_params)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/connection.py", line 119, in connect
    raise last_ex.with_traceback(None)
django.db.utils.OperationalError: connection failed: connection to server at "ip", port 5432 failed: session is read-only

[ngotzmann]

I fixed the problem with setting externalDatabase.targetSessionAttrs: any.
So my database configurations look like this:

externalDatabase:
  ...
  connMaxAge: 300
  disableServerSideCursors: true
  targetSessionAttrs: any