Skip to content

Commit f5b0fc2

Browse files
committed
Added sqlite-lib service to test against specific SQLite versions.
1 parent 385e14a commit f5b0fc2

File tree

4 files changed

+89
-3
lines changed

4 files changed

+89
-3
lines changed

.env

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,7 @@ MYSQL_VERSION=8.0
55
ORACLE_VERSION=23.5.0.0
66
POSTGRESQL_VERSION=14
77
POSTGIS_VERSION=3.1
8+
SQLITE_VERSION=3.31.0
9+
SQLITE_CFLAGS="-DSQLITE_ENABLE_DESERIALIZE \
10+
-DSQLITE_ENABLE_JSON1 \
11+
-DSQLITE_MAX_VARIABLE_NUMBER=32766"

README.md

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ $ docker compose run --rm mysql
6666
$ docker compose run --rm oracle
6767
$ docker compose run --rm postgres
6868
$ docker compose run --rm sqlite
69+
$ docker compose run --rm sqlite-lib
6970
```
7071

7172
Each of the above commands will run the test suite for a different supported
@@ -204,7 +205,14 @@ The versions of various backend services can be switched by setting these enviro
204205
| `ORACLE_VERSION` | `23.5.0.0` | Version of Oracle container image to use |
205206
| `POSTGRESQL_VERSION` | `14` | Version of PostgreSQL container image to use |
206207
| `POSTGIS_VERSION` | `3.1` | Version of PostGIS extension to use |
208+
| `SQLITE_VERSION` | `3.31.0` | Version of SQLite to compile and use |
207209

210+
> [!NOTE]
211+
>
212+
> Using a specific SQLite version requires compiling it from source. To
213+
> customize the `CFLAGS` used for the compilation, you can set the
214+
> `SQLITE_CFLAGS` environment variable. See the [`.env`][10] file for its
215+
> default value. For more details, see [SQLite Versions](#SQLite-Versions).
208216
209217
### Python Versions
210218

@@ -229,7 +237,8 @@ restrictions with respect to the range of versions available.
229237
### Database Versions
230238

231239
Most database container images are pulled from [Docker Hub][2]. Oracle database
232-
is pulled from the [Oracle Container Registry][3].
240+
is pulled from the [Oracle Container Registry][3]. Specific versions of SQLite
241+
are compiled directly from the tags in the [official Git mirror][11].
233242

234243
You can switch the version of the database you test against by changing the
235244
appropriate environment variable. Available options and their defaults can be
@@ -273,6 +282,41 @@ To determine what database versions can be used you can check the release notes
273282
for the branch of Django that you have checked out, or alternatively there is
274283
the [supported database versions][4] page on Django's Trac Wiki.
275284
285+
#### SQLite Versions
286+
287+
SQLite is normally bundled in the Python installation using the version
288+
available on the system where Python is compiled. We use the Python Docker image
289+
based on Debian `bookworm`, which has SQLite 3.40.1.
290+
291+
To use a different version, we compile SQLite from source and load the library
292+
dynamically using `LD_PRELOAD`. There are a few caveats as a result:
293+
294+
- Some SQLite features are only available if certain flags are set during
295+
compilation. SQLite is known to change these flags in newer releases, such as
296+
to enable features by default that were previously opt-in. When Python is
297+
compiled, it inspects the system's SQLite to determine features that are
298+
included in the `sqlite` module. A mismatch in the module and the dynamically
299+
loaded library may result in Python failing to load, which may happen if we
300+
use an SQLite version that is older than the system version.
301+
- Debian and Ubuntu use a custom `CFLAGS` variable to compile their distributed
302+
SQLite. Historically, Django's CI has only been configured with SQLite
303+
versions that come with the operating system. If SQLite is compiled with
304+
different flags, some tests may fail.
305+
306+
We currently work around the above caveats by setting the simplest `CFLAGS`
307+
value that allows all the tests to pass. In the future, the Django codebase may
308+
be more robust when tested against the different SQLite configurations and these
309+
workarounds may no longer be necessary.
310+
311+
Running the tests against a specific SQLite version must be done using the
312+
`sqlite-lib` container instead of `sqlite`.
313+
314+
```console
315+
$ docker compose run --rm sqlite-lib
316+
```
317+
318+
This is done to avoid compiling SQLite when you are not testing against a
319+
specific version.
276320

277321
### Other Versions
278322

@@ -309,3 +353,5 @@ with no promises that they'll be delivered:
309353
[7]: https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/unit-tests/#running-the-unit-tests
310354
[8]: https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/unit-tests/#running-the-selenium-tests
311355
[9]: https://docs.djangoproject.com/en/stable/ref/contrib/gis/testing/#geodjango-tests
356+
[10]: .env
357+
[11]: https://github.com/sqlite/sqlite

compose.yml

Lines changed: 37 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ x-base: &base
88
args:
99
- PYTHON_IMPLEMENTATION=${PYTHON_IMPLEMENTATION}
1010
- PYTHON_VERSION=${PYTHON_VERSION}
11-
additional_contexts:
11+
additional_contexts: &additional-contexts
1212
src: ${DJANGO_PATH:-../django}
1313
volumes:
1414
- ${DJANGO_PATH:-../django}:/django/source:rw
@@ -82,6 +82,36 @@ x-postgresql-base: &postgresql-base
8282
-c wal_level=minimal
8383
# 13+: -c wal_keep_size=0
8484

85+
x-sqlite-lib-base: &sqlite-lib-base
86+
image: django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION}-sqlite${SQLITE_VERSION}
87+
pull_policy: never
88+
build:
89+
context: .
90+
dockerfile_inline: |
91+
FROM django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION}
92+
SHELL ["/bin/bash", "-o", "errexit", "-o", "nounset", "-o", "pipefail", "-o", "xtrace", "-c"]
93+
# Only compile SQLite and set LD_PRELOAD if a version is specified.
94+
RUN <<EOF
95+
if [[ "${SQLITE_VERSION}" ]]; then
96+
export CFLAGS="${SQLITE_CFLAGS}"
97+
git clone --depth 1 --branch version-${SQLITE_VERSION} \
98+
https://github.com/sqlite/sqlite.git /tmp/sqlite
99+
cd /tmp/sqlite
100+
./configure
101+
make
102+
cp .libs/libsqlite3.so /tmp/
103+
rm -rf /tmp/sqlite
104+
fi
105+
EOF
106+
SHELL ["/bin/bash", "-c"]
107+
ENV LD_PRELOAD=${SQLITE_VERSION:+/tmp/libsqlite3.so}
108+
args:
109+
- PYTHON_IMPLEMENTATION=${PYTHON_IMPLEMENTATION}
110+
- PYTHON_VERSION=${PYTHON_VERSION}
111+
- SQLITE_VERSION=${SQLITE_VERSION}
112+
- SQLITE_CFLAGS=${SQLITE_CFLAGS}
113+
additional_contexts: *additional-contexts
114+
85115
x-memcached: &memcached-base
86116
image: memcached:alpine
87117
deploy:
@@ -283,13 +313,18 @@ services:
283313
- DATABASE_ENGINE=django.db.backends.postgresql
284314
- DATABASE_HOST=postgresql-db
285315

286-
sqlite:
316+
sqlite: &sqlite-base
287317
<<: *base
288318
depends_on:
289319
<<: *depends-on-caches
290320
environment:
291321
- DATABASE_ENGINE=django.db.backends.sqlite3
292322

323+
sqlite-lib:
324+
<<:
325+
- *sqlite-lib-base
326+
- *sqlite-base
327+
293328
# Commands: Tests: GIS
294329

295330
mariadb-gis:

packages.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,5 @@ libpq-dev
1414
libproj-dev
1515
libsqlite3-mod-spatialite
1616
pkg-config
17+
tcl-dev
1718
unzip

0 commit comments

Comments
 (0)