Docassemble packaged for Cloudron (self-hosting platform) - working v0.1

[OrcVole]

Summary

We have packaged docassemble as a custom application for Cloudron, a self-hosting platform that manages applications, databases, backups, and TLS on a single server. The package is working: interviews run, the Playground functions, documents generate, and data persists across restarts.

The full write-up, including step-by-step deployment instructions, architecture decisions, and a detailed list of gotchas we encountered, is on the Cloudron forum:

https://forum.cloudron.io/topic/15247/how-to-install-docassemble-on-cloudron-as-a-custom-application/2

Source repository: https://github.com/OrcVole/docassemble-cloudron

How it works

The package builds on top of the official jhpyle/docassemble Docker image and adapts it for Cloudron's managed environment:

• PostgreSQL and Redis are provided by Cloudron's managed addons (external to the container), connected via environment variables mapped to docassemble's config.yml.
• RabbitMQ is replaced by LavinMQ, an AMQP 0.9.1 wire-compatible broker that uses approximately 40 MB of RAM instead of 200 MB. Celery connects to it identically.
• Cloudron's reverse proxy handles TLS termination. The app runs with BEHINDHTTPSLOADBALANCER=true and nginx listening on port 8080.
• CONTAINERROLE is set to web:celery:cron:log:mail (not all) to prevent the init script from starting internal PostgreSQL and Redis.
• The read-only filesystem is handled by symlinking approximately 25 write paths to either /app/data/ (persistent) or /run/ (volatile, recreated each boot).

Where upstream changes could help

During packaging we identified a few areas where small upstream changes would make Cloudron deployment (and potentially other constrained-environment deployments) easier. These are not bugs, just observations from working through the process.

1. DAREADONLYFILESYSTEM coverage. The flag is checked in about 25 places in initialize.sh, which is great. However, several write operations are not gated by it (for example, writes to /etc/localtime, /etc/locale.conf, /etc/cron.daily/, /var/www/.certs, /usr/share/docassemble/webapp/syslog-ng-orig.conf). Extending the flag's coverage would reduce the number of symlinks needed for read-only environments.

2. initialize.sh does not exit after completing. After printing "Finished initializing" and starting all services, the initialize process continues running indefinitely. In our setup it consumes memory without doing further work. If this is intentional (for example, to keep supervisor from restarting it), a comment explaining why would be helpful.

3. initialize.sh force-starts postgres via supervisorctl even when CONTAINERROLE does not include sql and even when DBHOST is set to an external server. On earlier attempts we had CONTAINERROLE=all, and the script started the internal PostgreSQL despite our supervisor override setting autostart=false. Switching to CONTAINERROLE=web:celery:cron:log:mail resolved this.

4. Connection to postgres system database during init. The init script tests if the app database exists by connecting to the postgres database. Some managed PostgreSQL providers (including Cloudron's) only allow connections to the app-specific database, so this check fails. It is non-fatal (the script proceeds to create tables on the correct database), but generates confusing error messages in the logs.

Known limitations of the Cloudron package

• First boot takes 15-20 minutes (venv copy, database migration, package table population)
• Exim4 (email receiving) does not work due to read-only /etc/exim4/; outbound email works via Cloudron's SMTP addon
• Playground package installs persist across restarts but not across image updates
• No OIDC/SSO integration with Cloudron's user management yet
• Requires ~8 GB RAM allocated to the container

Not a feature request or bug report

This is purely informational. We wanted to let the docassemble community know that a Cloudron deployment path exists and is working, and to share our findings in case they are useful for improving constrained-environment support. The upstream observations above are minor and the existing Docker support is excellent. The BEHINDHTTPSLOADBALANCER, DAREADONLYFILESYSTEM, and external database/Redis support made this packaging work possible.

Thank you for building and maintaining docassemble.

[jhpyle]

Very cool. I will look into LavinMQ.

If your platform supports Docker Compose, you might want to look at https://github.com/jhpyle/docassemble-compose, which shows how you can cut out the jhpyle/docassemble image, supervisord, and any service you are hosting externally.

[OrcVole]

From the discussion in the Cloudron forum:

"The strongest path would be a hybrid approach, similar to how the Baserow Cloudron package evolved. There are two realistic strategies, and they have different tradeoffs.

Strategy 1: Upstream-first (recommended)

Work with Jonathan Pyle (docassemble maintainer) to improve the upstream Docker image's support for constrained environments. The specific changes that would make official Cloudron packaging dramatically easier:

• First, extend DAREADONLYFILESYSTEM to cover all write paths, not just the 25 it currently gates. If the upstream image could run cleanly with that flag set to true and all state under a single configurable data directory, the Cloudron package would be straightforward.

• Second, make initialize.sh respect CONTAINERROLE fully. If sql is not in the role, the script should never touch PostgreSQL, not even to check if it is running. Same for Redis and RabbitMQ.

• Third, support a DA_DATA_DIR environment variable that relocates all writable state to a single directory. Docassemble already partially supports this with DA_ROOT, but many paths are hardcoded outside of it (/var/www/.certs, /etc/ssl/docassemble, /var/lib/nginx, etc).

If those three changes landed upstream, an official Cloudron package could be built on cloudron/base:5.0 with a clean Dockerfile that installs docassemble from pip into a venv, uses Cloudron addons for PostgreSQL and Redis, and needs very few symlink hacks.
"

[OrcVole]

Very cool. I will look into LavinMQ.

If your platform supports Docker Compose, you might want to look at https://github.com/jhpyle/docassemble-compose, which shows how you can cut out the jhpyle/docassemble image, supervisord, and any service you are hosting externally.

Thank you for pointing us to docassemble-compose. This is exactly what we need for the next iteration.

Our v0.1 builds on top of the monolithic jhpyle/docassemble image, which meant we spent most of our effort working around initialize.sh and the read-only filesystem. We ended up with approximately 25 symlinks to make various write paths writable, supervisor overrides to disable internal PostgreSQL/Redis/RabbitMQ, and a workaround for the initialize process never exiting after completion. It works, but it is fragile.

The compose approach eliminates nearly all of that. With Dockerfile.core building a lean image from Ubuntu directly, and run.sh/run-www.sh replacing supervisord and initialize.sh, the problems we fought with simply do not exist:

• No supervisord to override
• No initialize.sh hanging after completion
• No internal services to disable
• No CONTAINERROLE confusion
• Far fewer filesystem paths that need write access
• Smaller image

Cloudron does not support Docker Compose natively (each app is a single container), but the compose repo gives us the individual service definitions and startup scripts that we can combine into one clean container. The plan for v0.2 is:

1. Use Dockerfile.core as the starting point
2. Add LavinMQ as the internal AMQP broker (we confirmed this works as a drop-in for RabbitMQ in v0.1, saves ~160 MB RAM)
3. Wire Cloudron's managed PostgreSQL and Redis addons via environment variables mapped to config.yml
4. Adapt run.sh to handle Cloudron's reverse proxy setup (BEHINDHTTPSLOADBALANCER, port 8080)
5. Handle the writable venv for Playground package installs

This should be a much cleaner package. We will update the repository and the Cloudron forum thread when v0.2 is ready.

Thanks again for maintaining docassemble and for the responsive guidance. The existing support for external databases, external Redis, and behind-proxy operation made the initial packaging work possible, and the compose repo makes the path forward much clearer.

[OrcVole]

We are going to try a version 2, now that we have the proof-of-concept. jhpyle, we have some decisions to make and wonder whether you might have some input / help us decide. Which would you choose? Our main goal for version 2 would be to help make any attempt at a version 3 (eg fully supported on Cloudron) easier. We are therefor tempted to go for the bigger options.

Decision Points Before Building

1. Is Pandoc/TeX required for v0.2? If the pilot interviews only output simple documents, we could defer it and ship a much smaller image. If document assembly to PDF is core, we must include it.

2. Is DOCX to PDF conversion required for v0.2? This determines whether Gotenberg (or an alternative) must be in scope.

3. Should the admin account be provisioned automatically on first boot? The compose repo supports this via /run/secrets/initial-creds. We could provision from Cloudron environment variables or require manual setup through the web interface.

4. What memory limit should be documented? If Pandoc is included, 8 GB is safest. If not, 4 GB may suffice for normal operation.