PDF export via the Java worker fails end-to-end in the local stack (five sequential defects)

[scaphilo]

Triggering Create PDF on an Invoice never produced a downloadable document. Fixing it surfaced five independent defects, each hidden behind the previous one. Four of the five only manifest locally, because they all come down to the same root theme: container-internal endpoint names and identities leaking into places that assume public, browser-reachable, or cloud-shaped values.

This issue records the whole chain; the fix spans three repos (koalixcrm, koalixcrm-system, koalixcrm-fop-service) and must be deployed together.

---

1. Worker rejected workspace_id — stale image, not missing code

UnrecognizedPropertyException: Unrecognized field "workspace_id" (class net.koalix.pdf.sqs.PdfExportCommand),
not marked as ignorable (5 known properties: ...)

PdfExportCommand.java already declared workspace_id; the running jar predated it. docker-compose.yml pulled koalixcrm_pdf_service:${PDF_SERVICE_IMAGE_TAG:-develop} with no pull_policy, so Compose reused a locally cached image of a moving tag indefinitely.

Fix: pdf-service now builds from the koalixcrm-fop-service working tree (KOALIXCRM_FOP_DIR) instead of pulling from GHCR, so the worker always matches the checked-out sources. PDF_SERVICE_IMAGE_TAG is gone.

2. OIDC M2M returned 401 invalid_client

secrets.env held a chimera: issuer from the Digital Company admin realm (digital-company-dev), a client id that exists in no repo config (workflow-support-m2m-celery-worker), and a scope in AWS Cognito resource-server syntax (m2m/read, from aws_cognito_resource_server "m2m" in the identity terraform). WFS's own M2M client is wfs-celery-m2m in realm quantalq-dev.

Fix (ops): a koalixCRM-owned confidential client with service accounts enabled. No code change, but the diagnosis was invisible because the worker logged only the status code — see 3.

3. Scope handling diverged from WFS — 400 invalid_scope

Once the client was valid, Keycloak rejected the scope. WFS's base_api_client.py never sends scope; koalixCRM's api_client.py sent it conditionally, and the Java port inherited that.

Fix: scope removed end-to-end (Java OidcTokenProvider / AppProperties / application.yaml, Python api_client.py, settings, secrets.env.example) so both services build the identical three-field payload. The token request and discovery now also surface the response body, which carries OAuth's error_description — without it a 400/401 here is undiagnosable, and each round of guessing cost a rebuild.

4. Template assets were never uploaded — 404 templates/xsl/invoice.xsl

minio-setup only ran mc mb; nothing ever uploaded the DocumentTemplate files. The bucket held zero objects while the DB carried 10 rows pointing into it (names inherited from the filesystem-era install).

Fix: new management command seed_document_template_files, wired into docker/dev/entrypoint.sh after migrate. Idempotent, never writes to the DB. Matches by basename, falling back to the name with Django's 7-char collision suffix stripped (fop_config/fontconfig_tOYVu30.xml → repo's fontconfig.xml). All 30 file references in the dev DB resolve.

5. Status write-back rejected — 400 on PATCH .../pdf-export-processes/{id}/

PDFExportProcess.result_url was a URLField. Django's URLValidator requires a TLD or literal localhost, so it rejects http://minio:9000/.... The PDF had already rendered and uploaded successfully — only the write-back failed.

Telling detail: the worker POSTs CommercialDocumentMedia with the same URL immediately before, and that succeeded — because s3_url there was already a CharField. Two models carrying the same value, typed differently.

Fix: result_url → CharField (migration core/0010). Column type unchanged; validators only.

---

Follow-on: port S3Media from WFS

The result_url fix treats the symptom. The root cause is storing absolute URLs, which bakes the endpoint host into the row. WFS already solved this (ADR-7 / QUAQ2-110): S3Media.s3_url stores a relative object key, with a validator that explicitly rejects http://minio:9000/bucket/key.

Ported into koalixcrm.core:

• core/models/s3_media.py — abstract S3Media + validate_s3_url_is_relative. Adapted: WFS models media_type as an FK to a MediaType table; koalixCRM keeps its MIME CharField.
• CommercialDocumentMedia → CommercialDocumentS3Media(S3Media, WorkspaceScopedModel); s3_key dropped, s3_url now holds the key it used to duplicate. Schema-only migration (contracts/0018), db_table unchanged.
• Java CommercialDocumentMediaDto sends the relative key. Breaking pair — the validator rejects what the old worker sends, so both sides deploy together.

Follow-on: presigned downloads in the admin

There was no way to download a generated PDF. The existing download_link on the media admin was doubly broken: settings.S3_MEDIA_BUCKET was undefined (an AttributeError the except ClientError didn't catch, so the page 500'd), and get_s3_client always took the S3_ENDPOINT_URL branch, making use_presigned_config dead code and embedding minio:9000 in the URL.

SigV4 signs the Host header, so the host cannot be rewritten after signing — the client must be built against the public endpoint up front. Ported WFS's dual-endpoint pattern (ADR-019):

• get_s3_presign_client() + S3_PUBLIC_ENDPOINT_URL (falls back to the internal endpoint, so production needs no config).
• core/admin/s3_media_download.py — presigned_url_for_key, download_link_html, S3MediaDownloadMixin, s3_key_from_url for legacy absolute values. Placed in core rather than contracts (WFS's location) because PDFExportProcess needs it and core must not depend on contracts.
• Download links now on PDF Export Processes, the CommercialDocument inline (Invoice/Quotation/…), and a new collapsed Generated documents panel on Contract.

CommercialDocumentS3Media has no FK to Contract, so the Contract view is an aggregated read-only list rather than an inline.

Also fixed along the way

PDFExportProcess was registered in the admin but absent from the Grappelli dashboard, so the "N job(s) queued — check PDF Export Processes" messages pointed at a page with no navigation to it. Added a dashboard entry, made the message a link (core/pdf_export_messages.py, 6 call sites), and added a result column.

Known-stale, not addressed

koalixcrm_install_defaulttemplates is dead code — it references djangoUserExtension.models.XSLFile, TemplateSet.invoiceXSLFile and filebrowser.FileObject, all pre-2.0 API. It would fail on import and creates new rows rather than repairing existing ones. Worth its own issue.

Verification status

Lint-clean with no regressions; s3_key_from_url and the seed command's name resolution were validated directly against the dev DB; a test asserting the validator rejects absolute URIs was added. The full stack has not been run against these changes — no Django check, no test suite, no Java compile were possible in the authoring environment.