Skip to content

gh-137205: Document how to safely use PRAGMA during SQLite transactions #137505

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

gh-137205: Document how to safely use PRAGMA during SQLite transactions #137505

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
a637bde
gh-137205: Document how to safely use PRAGMA during SQLite transactions
erlend-aasland Aug 7, 2025
9b0e56e
conn => con
erlend-aasland Aug 7, 2025
b525dd8
Whitespace
erlend-aasland Aug 7, 2025
ae5d700
Promote the new howto in the autocommit attr reference
erlend-aasland Aug 7, 2025
fc2e957
Fix doctest
erlend-aasland Aug 7, 2025
5ef2d72
Add missing semicolon to seealso
erlend-aasland Aug 7, 2025
155013f
Correctly setup and format example doctest code
erlend-aasland Aug 7, 2025
345043f
Amend How-to heading, and remove a recommendation
erlend-aasland Aug 11, 2025
000e4c5
Mention layday's suggestion
erlend-aasland Aug 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 55 additions & 7 deletions Doc/library/sqlite3.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,7 @@ inserted data and retrieved values from it in multiple ways.
* :ref:`sqlite3-converters`
* :ref:`sqlite3-connection-context-manager`
* :ref:`sqlite3-howto-row-factory`
* :ref:`sqlite3-howto-pragma-in-transaction`

* :ref:`sqlite3-explanation` for in-depth background on transaction control.

Expand Down Expand Up @@ -1353,8 +1354,6 @@ Connection objects
implying that :mod:`!sqlite3` ensures a transaction is always open.
Use :meth:`commit` and :meth:`rollback` to close transactions.

This is the recommended value of :attr:`!autocommit`.

* ``True``: Use SQLite's `autocommit mode`_.
:meth:`commit` and :meth:`rollback` have no effect in this mode.

Expand All @@ -1367,7 +1366,10 @@ Connection objects
Changing :attr:`!autocommit` to ``False`` will open a new transaction,
and changing it to ``True`` will commit any pending transaction.

See :ref:`sqlite3-transaction-control-autocommit` for more details.
.. seealso::

* :ref:`sqlite3-transaction-control-autocommit`
* :ref:`sqlite3-howto-pragma-in-transaction`

.. note::

Expand Down Expand Up @@ -2429,6 +2431,39 @@ the context manager does nothing.
couldn't add Python twice


.. _sqlite3-howto-pragma-in-transaction:

How to use ``PRAGMA`` statements with implicit transaction control
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Some ``PRAGMA`` statements have no effect within transactions.
For example, a ``PRAGMA foreign_keys=ON`` query will silently fail
if there is an open transaction.
In order to safely use ``PRAGMA`` statements,
ensure that there is no open transaction:

* If :attr:`~Connection.autocommit` is ``True``
and a transaction was explicitly opened,
make sure to :meth:`~Cursor.execute` a ``COMMIT``
before executing the ``PRAGMA`` statement.
* Else, temporarily disable implicit transaction control
using the :attr:`!autocommit` attribute
before executing the ``PRAGMA`` statement:

.. testcode::
:hide:

con = sqlite3.connect(":memory:", autocommit=False)

.. testcode::

cur = con.cursor()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wish the state of affairs was better here, but while I can think of various API features that would make this less klunky, I dont think any of them would be worth it, so here we are.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think a decorator is worth it. Since PRAGMAs are mostly used during the setup of a database connection, I think @layday's approach is the most sane: connect using autocommit=False, execute your PRAGMAs, then set the autocommit attr to whatever suits your application best.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my API features would be either:

  1. a context manager
with conn.autocommit_block:
    # ...
  1. a PRAGMA method
   cursor.pragma("FOREIGN KEYS=ON")
  1. include commonly needed pragmas as part of connect() . A bitwise enum is sort of nice here but doesnt work for pragmas that have numeric arguments
   # doesnt work for every kind of pragma but looks nice
   conn = sqlite3.connect(...., pragmas=Pragma.FOREIGN_KEYS_ON | Pragma.JOURNAL_MODE_PERSIST)

in your comment I assume you meant "connect with autocommit=True".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. It's a nice idea, but we already have a context manager for transactions. I'm not sure it is worth it to introduce a context manager for the opposite.
  2. & 3: I would prefer to not add more APIs with side effects. Moreover, I would prefer to not manage an enum of supported PRAGMAs.

[...] in your comment I assume you meant "connect with autocommit=True".

Yes, thanks.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a few things I don't like about the proposed workaround:

  • It's rather verbose. Pragmas are typically only set once on connect, so it's sufficient to initialise connect with autocommit=True before flipping it off.
  • It has the appearance of a bodge, though it's decidedly less bad than manually executing a ROLLBACK or COMMIT followed by a BEGIN.
  • Entering autocommit implicitly emits a COMMIT, which has a 'spooky action at a distance' feel to it.
  • Enabling foreign keys is more important than any Python-specific recommendation made by the docs, and if autocommit=False doesn't support the FK pragma out of the box, that vastly reduces its usefulness.

I don't really know what'd work better, but I'm gravitating towards an init hook that'd be executed before the first BEGIN, mirroring SQLA's listens_for idiom. I don't think that's in the scope of this PR though, so I'd personally be happy if the suggestion was rephrased to:

Suggested change
cur = con.cursor()
* Else, initialize the connection with ``autocommit=True``
and set the :attr:`!autocommit` attribute to ``False``
after executing the ``PRAGMA`` statement:
.. testcode::
con = sqlite3.connect(":memory:", autocommit=True)
cur.execute("PRAGMA foreign_keys = ON")
con.autocommit = False # Enable implicit transaction control.

a context manager

I don't think an autocommit context manager is generally useful, and I can see it being misused to break out of a transaction for any number of reasons, e.g. to fetch changes made by another connection with WAL on.

a PRAGMA method

I don't know how this would work, but SQLite has pragmas which have no effect outside of a transaction - defer_foreign_keys is one of them. If the idea is that pragma(...) would exit a transaction to set the provided pragma, then the driver's gonna have to encode specific knowledge of SQLite pragmas.

saved = con.autocommit
con.autocommit = True # Disable implicit transaction control.
cur.execute("PRAGMA foreign_keys=ON")
con.autocommit = saved # Restore the previous setting.


.. _sqlite3-uri-tricks:

How to work with SQLite URIs
Expand Down Expand Up @@ -2643,11 +2678,9 @@ Transaction control via the ``autocommit`` attribute

The recommended way of controlling transaction behaviour is through
the :attr:`Connection.autocommit` attribute,
which should preferably be set using the *autocommit* parameter
of :func:`connect`.
normally set using the *autocommit* parameter of :func:`connect`.

It is suggested to set *autocommit* to ``False``,
which implies :pep:`249`-compliant transaction control.
Set *autocommit* to ``False`` for :pep:`249`-compliant transaction control.
This means:

* :mod:`!sqlite3` ensures that a transaction is always open,
Expand All @@ -2660,6 +2693,16 @@ This means:
* An implicit rollback is performed if the database is
:meth:`~Connection.close`-ed with pending changes.

.. note::

Some ``PRAGMA`` statements have no effect when a transaction is open.
If ``PRAGMA`` statements are a part of the database connection setup
and you want :pep:`249`-compliant transaction control,
set *autocommit* to ``True`` in :func:`connect`,
execute the needed ``PRAGMA`` statements,
then set :attr:`Connection.autocommit` to ``False``.
See :ref:`sqlite3-howto-pragma-in-transaction` for details.

Set *autocommit* to ``True`` to enable SQLite's `autocommit mode`_.
In this mode, :meth:`Connection.commit` and :meth:`Connection.rollback`
have no effect.
Expand Down Expand Up @@ -2716,6 +2759,11 @@ The :meth:`~Cursor.executescript` method implicitly commits
any pending transaction before execution of the given SQL script,
regardless of the value of :attr:`~Connection.isolation_level`.

.. note::

Some ``PRAGMA`` statements cannot be set when a transaction is open.
See :ref:`sqlite3-howto-pragma-in-transaction` for details.

.. versionchanged:: 3.6
:mod:`!sqlite3` used to implicitly commit an open transaction before DDL
statements. This is no longer the case.
Expand Down
Loading