Build container and release it to github, merge files to pyproject.toml and handle basic setup/migrations automatically in container entrypoint-script by ilkka-ollakka · Pull Request #3636 · bookwyrm-social/bookwyrm

ilkka-ollakka · 2025-07-06T12:16:54Z

Description

This PR contains changes to build bookwyrm container and release it to be used without closing the source repository. Currently it builds arm64 and amd64 platforms.

Container builds are done when tag is pushed to use tag as version, for example bookwyrm:0.8.0. Main repo push build container with name bookwyrm:main.

It also combines requirements.txt, pylint/mypy/coverage configs to pyproject.toml config.

Container entrypoint script runs migrations,initdb and permissions checks before changing to bookwyrm-user to start provided command. So Celery/bookwyrm services are running non-root user. I used the #2467 as initial entrypoint script and modified it littlebit.

bw-dev command now has command dev to start local code in container instead of build container. It should allow local development easily. dev-config also sets DEBUG=1 environment variable, making heavy assumption that you want to run it with debug-mode.

One flow is not yet handled properly, that is that celery doesn't wait migrations after initial migrations, so new migrations are done, but celery/others doesn't know to wait them to finish before starting.

I modified Dockerfile and entrypoint for this, to take build-timestamp as marker what to look for in file when waiting migration checks, still might not be fully working, but should handle new builds/migrations to be waited correctly.

Changes include using dev-tools container changes, build it on top of bookwyrm container, install all dev-dependencies there instead of main container and run all pytest/mypy etc using that container.

Related Issue building / publishing generic Docker images #2095
Closes Consider packaging releases as containers #3551

What type of Pull Request is this?

Bug Fix
Enhancement
Plumbing / Internals / Dependencies
Refactor

Does this PR change settings or dependencies, or break something?

This PR changes or adds default settings, configuration, or .env values
This PR changes or adds dependencies
This PR introduces other breaking changes

Details of breaking or configuration changes (if any of above checked)

We don't anymore mount local directory to container, but build container. We also run migrations and database-initdb autotically on each start.

We run pytest/mypy/etc in derived container that mounts source-code in.

Dev flow now uses bw-dev dev instead of bw-dev up to start local code.

requirements.txt, .pylintrc, .coveragerc and mypy.ini are all merged to pyproject.toml

Documentation

New or amended documentation will be required if this PR is merged
I have created a matching pull request in the Documentation repository
I intend to create a matching pull request in the Documentation repository after this PR is merged

Tests

My changes do not need new tests
All tests I have added are passing
I have written tests but need help to make them pass
I have not written tests and need help to write them

I think this needs some manual testing, as it is big change and it is probable that I have missed some flow/use-case that is broken.

ilkka-ollakka · 2025-07-14T12:09:16Z

rebased the PR on top of current main

ilkka-ollakka · 2025-07-14T13:50:46Z

If there is something that you are not sure/just curious, please do ask and I'll try to clarify what it does.

Dockerfile

ilkka-ollakka · 2025-07-15T10:10:38Z

I'll add commits to this PR to have development flow working with local source-files.

ilkka-ollakka · 2025-08-06T18:02:19Z

I moved dev-tools to dev-compose file and point docker-compose bookwyrm images to ghcr.io, so release-wise the docker-compose modifications would be just adding some production related db-changes and change bookwyrm:main to bookwyrm:release-number.

marcusyoung · 2025-08-06T23:02:50Z

@ilkka-ollakka Can I check that consideration has been given to any potential impact of these changes on dockerless installs. For example, I noticed that requirements.txt has been removed, But we need to run ./venv/bin/pip3 install -r requirements.txt on a dockerless install during upgrade. There may be other things, but that's the one that stood out.

ilkka-ollakka · 2025-08-07T06:56:50Z

@ilkka-ollakka Can I check that consideration has been given to any potential impact of these changes on dockerless installs. For example, I noticed that requirements.txt has been removed, But we need to run ./venv/bin/pip3 install -r requirements.txt on a dockerless install during upgrade. There may be other things, but that's the one that stood out.

Ah yes, good point. For now it should be something like pip install "pip>=25.1.0" && pip install --group main . so to have new enough pip to support group and it should be able to use the pyprojet.toml-file just fine.

ilkka-ollakka · 2025-08-09T08:26:13Z

Rebased on top of current main and update pyproject.toml with new Django-version from updated requirements.txt.

ilkka-ollakka · 2025-08-24T12:07:41Z

rebased on top of current main with updated requirements.txt changes.

ilkka-ollakka · 2025-09-17T14:56:01Z

rebased on top of main to update requirements changes

ilkka-ollakka · 2025-09-19T16:34:30Z

I'll split separate PR's out from this, separatable changes so those are easier to review/merge separately after current release is in order.

hughrun · 2025-10-11T23:41:35Z

@ilkka-ollakka I'm trying to understand this. Can you explain how entrypoint.sh gets called? I can't see it referenced anywhere. I'm also wondering if we can use depends_on with a (maybe simpler?) health check instead?

ilkka-ollakka · 2025-10-12T06:44:12Z

@ilkka-ollakka I'm trying to understand this. Can you explain how entrypoint.sh gets called? I can't see it referenced anywhere. I'm also wondering if we can use depends_on with a (maybe simpler?) health check instead?

Ah yes, in Dockerfile we define 'ENTRYPOINT' pointing to that script in line 42. So all commands get run as parameter to entrypoint script.

Documentation on the difference of ENTRYPOINT and CMD can be found in https://docs.docker.com/reference/dockerfile/#understand-how-cmd-and-entrypoint-interact so docker/podman itself handles the calling

I think it is good idea use health checks to make entrypoint simpler.

Or did you have some idead to use healtchecks instead entrypoint script? That would be also nice approach, and if you have something started with that flow, I would be happy to take a look and adjust the PR?

otherwise it would shadow installed node_modules location

…ations Instead of checking file-presence, check that file contains build-time define hash of migration files This allows newer builds/migrations to be waited until those are checked properly

docker-compose.dev.yml mounts most of the files on top of container, so you can develop entrypoint.sh as well as bookwyrm code/locales etc with dev command.

we only expose that port in web role, not in celery etc containers

…ypoint Celery and flower containers wait until web container is healthy, so it has handled migrations and database init This allows to simplify entrypoint-script so it doesn't need to poll in other containers if migrations are done

ilkka-ollakka · 2025-10-12T13:33:38Z

I modified the docker-compose to have depends_on to web-container being healthy on bookwyrm containers and defined healthcheck to web-container to check that django is up. Also cleaned out entrypoint to run migrations and not to do polling in other containers.

ilkka-ollakka · 2025-10-12T13:33:49Z

also rebased on top of curren main

SensibleSalmon · 2026-01-12T01:15:10Z

Hi all, very happy to see this as someone looking to deploy this on a swarm. Is there any additional testing or development that I could help with on this? I see #3759 is approved and passing build; I'm happy to jump in wherever is useful. I think a canonical docker image is a fantastic addition to the project.

ilkka-ollakka · 2026-01-12T06:20:51Z

Hi all, very happy to see this as someone looking to deploy this on a swarm. Is there any additional testing or development that I could help with on this? I see #3759 is approved and passing build; I'm happy to jump in wherever is useful. I think a canonical docker image is a fantastic addition to the project.

Thats nice to hear! I've been splitting this PR to more bite-size PRs and I think only things left currently are:

build full container image and publish it, most likely I'll split this into 2 separate PR
run container as non-root user
dev-tools/etc fiddling

I most likely try to split those remaining things during this or next week to new PRs, and if you can take a look and test/comment, it would be great. Ofcourse if you have some questions/comments already now in things, I would love to hear those.

SensibleSalmon · 2026-01-13T22:15:35Z

Ofcourse if you have some questions/comments already now in things, I would love to hear those.

Sure thing! My docker compose experience is mildly limited; I have ~10 stacks running across either my NAS or my swarm and while my employer uses containers heavily I'm a step removed from the infra teams. I put myself firmly in the "know enough to be dangerous" category 😅 so just be aware that my questions are just as likely to come from a lack of (beginner) knowledge as being a structural concern, and I'll try to flag them accordingly.

I'll add what questions I have against the PR and I'll see if I can get an instance running on my swarm with the compose file in this PR / the ghcr.io/bookwyrm/bookwyrm:main image, though it might be the weekend before I have the time to do that.

SensibleSalmon · 2026-01-13T22:16:12Z

docker-compose.yml

      - main
  web:
+    image: ghcr.io/bookwyrm/bookwyrm:main
    build: .


curiosity: for the other services in the compose file you removed the build directive in favor of the image: ghcr.io/bookwyrm/bookwyrm:main; why keep the build: . here?

allowing development and local build. Also as all have same image it only builds it once, atleast podman compose builds currently all images separately.

SensibleSalmon · 2026-01-13T22:20:44Z

pyproject.toml

+    "ALLOWED_HOSTS = your.domain.here",
+    "BOOKWYRM_DATABASE_BACKEND = postgres",
+    "MEDIA_ROOT = images/",
+    "CELERY_BROKER_URL = memory://",


curiosity: is this meant to be a default? in the docker-compose case, would memory:// ever work when the broker is a separate container?

this is oblyfor pytest run variables, so not default otherwise. It is in tool pytest.ini_options block

SensibleSalmon · 2026-01-13T22:22:37Z

entrypoint.sh

+    chown -c -R bookwyrm /app/exports
+    chown -c -R bookwyrm /app/images
+    chown -c -R bookwyrm /app/static/css/themes


curiosity: what happens when the bookwyrm user doesn't exist on the host system? Docker user/group mapping is not something I understand well, and I feel like I often run into filesystem permissions errors.

it doesnt need to exist in host system. bookwyrm account is created inside container in build phase. Entrypoint is also runned inside the container in container start.

SensibleSalmon · 2026-01-17T15:49:09Z

docker-compose.yml

+        condition: service_healthy
+      redis_broker:
+        condition: service_started
    restart: on-failure


this conflicts with the restart: always directive on line 105, above

SensibleSalmon · 2026-01-17T15:49:41Z

docker-compose.yml

+    restart: always
    depends_on:
      - celery_worker
    restart: on-failure


another pair of conflicting restart directives

SensibleSalmon · 2026-01-17T15:49:56Z

docker-compose.yml

+    restart: always
    depends_on:
-      - db
-      - redis_broker
+      db:
+        condition: service_healthy
+      redis_broker:
+        condition: service_started
+      web:
+        condition: service_healthy
    restart: on-failure


and another

good catches, I'll try to remember to sort those out when I rebase/split remaining things out.

hughrun · 2026-01-30T04:10:52Z

@ilkka-ollakka is the idea that we won't actually need this PR once we merge #3803 #3809 plus maybe some small cleanup after that?

ilkka-ollakka · 2026-01-30T04:47:37Z

@ilkka-ollakka is the idea that we won't actually need this PR once we merge #3803 #3809 plus maybe some small cleanup after that?

yes, idea is to split things to smaller PRs and then this can be closed.

ilkka-ollakka · 2026-01-30T11:33:48Z

@hughrun just making sure, there are still few PRs that are needed beside those that are already open. Mainly the publish change is one and the non-root/dev-tools thing is optional but nice to have.

ilkka-ollakka force-pushed the feat/container_build_and_release branch from 9afc7eb to f6ce2c5 Compare July 6, 2025 12:23

ilkka-ollakka force-pushed the feat/container_build_and_release branch from bebcb12 to fbad6a6 Compare July 14, 2025 12:09

ilkka-ollakka changed the title ~~Build container and release it to github, merge files to pyproject.toml~~ Build container and release it to github, merge files to pyproject.toml and handle basic setup/migrations automatically in container entrypoint-script Jul 14, 2025

ilkka-ollakka commented Jul 15, 2025

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

ilkka-ollakka force-pushed the feat/container_build_and_release branch 2 times, most recently from f9e0fc1 to e816a01 Compare July 17, 2025 14:28

ilkka-ollakka force-pushed the feat/container_build_and_release branch 2 times, most recently from 4cd8fd9 to 9908401 Compare August 6, 2025 18:01

ilkka-ollakka force-pushed the feat/container_build_and_release branch from 9908401 to e403bf8 Compare August 9, 2025 08:19

ilkka-ollakka force-pushed the feat/container_build_and_release branch 3 times, most recently from e7f3c72 to 2afbcfa Compare August 15, 2025 16:29

hughrun added deployment config-change PR that includes configuration changes labels Aug 22, 2025

ilkka-ollakka force-pushed the feat/container_build_and_release branch from 2afbcfa to f8f2277 Compare August 24, 2025 12:07

ilkka-ollakka force-pushed the feat/container_build_and_release branch from f8f2277 to 98e27b3 Compare September 17, 2025 14:55

ilkka-ollakka mentioned this pull request Sep 19, 2025

change init_db command so it can be run against initialized database #3696

Merged

14 tasks

ilkka-ollakka mentioned this pull request Sep 19, 2025

merge settings and requirements files to pyproject.toml #3697

Merged

14 tasks

ilkka-ollakka and others added 7 commits October 12, 2025 16:32

fix dev-tools mounts

f658ad2

otherwise it would shadow installed node_modules location

Dockerfile: add buildtime hash of migrations as info to wait for migr…

fa53c6a

…ations Instead of checking file-presence, check that file contains build-time define hash of migration files This allows newer builds/migrations to be waited until those are checked properly

bw-dev: add 'dev' command to run local code instead of build container

41f3b8f

docker-compose.dev.yml mounts most of the files on top of container, so you can develop entrypoint.sh as well as bookwyrm code/locales etc with dev command.

docker-compose: change bookwyrm image to point to ghcr.io

3355663

docker-compose: move dev-tools container to dev-compose-file

d3b36aa

Dockerfile: remove expose port

2b7298b

we only expose that port in web role, not in celery etc containers

ilkka-ollakka force-pushed the feat/container_build_and_release branch from c806b70 to 73e3d43 Compare October 12, 2025 13:32

entrypoint: add missed compile_themes step

8ec41ff

SensibleSalmon reviewed Jan 13, 2026

View reviewed changes

SensibleSalmon reviewed Jan 17, 2026

View reviewed changes

ilkka-ollakka mentioned this pull request Jan 18, 2026

build full bookwyrm container #3803

Merged

14 tasks

ilkka-ollakka mentioned this pull request Jan 31, 2026

Dockerfile: mount cache for apt/npm/pip to improve rebuild times #3815

Merged

14 tasks

ilkka-ollakka mentioned this pull request Feb 10, 2026

Add github action to build and release containers #3841

Open

14 tasks

ilkka-ollakka marked this pull request as draft March 10, 2026 06:40

Uh oh!

Conversation

ilkka-ollakka commented Jul 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What type of Pull Request is this?

Does this PR change settings or dependencies, or break something?

Details of breaking or configuration changes (if any of above checked)

Documentation

Tests

Uh oh!

ilkka-ollakka commented Jul 14, 2025

Uh oh!

ilkka-ollakka commented Jul 14, 2025

Uh oh!

Uh oh!

ilkka-ollakka commented Jul 15, 2025

Uh oh!

ilkka-ollakka commented Aug 6, 2025

Uh oh!

marcusyoung commented Aug 6, 2025

Uh oh!

ilkka-ollakka commented Aug 7, 2025

Uh oh!

ilkka-ollakka commented Aug 9, 2025

Uh oh!

ilkka-ollakka commented Aug 24, 2025

Uh oh!

ilkka-ollakka commented Sep 17, 2025

Uh oh!

ilkka-ollakka commented Sep 19, 2025

Uh oh!

hughrun commented Oct 11, 2025

Uh oh!

ilkka-ollakka commented Oct 12, 2025

Uh oh!

ilkka-ollakka commented Oct 12, 2025

Uh oh!

ilkka-ollakka commented Oct 12, 2025

Uh oh!

SensibleSalmon commented Jan 12, 2026

Uh oh!

ilkka-ollakka commented Jan 12, 2026

Uh oh!

SensibleSalmon commented Jan 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hughrun commented Jan 30, 2026

Uh oh!

ilkka-ollakka commented Jan 30, 2026

Uh oh!

ilkka-ollakka commented Jan 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ilkka-ollakka commented Jul 6, 2025 •

edited

Loading