Feature/m8 f 159 cleanup ticket (#88)

* M8F-89: [Feature] Update docker files to setup and run the application correctly

* M8F-89: [Feature] Removed unnecessary migrate script

* Docker setup completed

* Dockeker working

* Create a service that initializes the keycloak realms

* Removed unnecessary logic from backend

* M8F-89: [Feature] Updated docker compose to run keycloak realm setup on start

* User non root user in docker

* M8F-89: [Feature] Updated document and removed unnecessary script

* M8F-89: [Feature] fix: Keycloak theme CLI opts; add healthcheck and backend depends_on keycloak healthy

* M8F-89: [Feature] Run images with non-root default user

* Docker Deployed

* Deployment workking

* Cleanup logs and minor optimization

* M8F-89: [Feature] Updated tenant creation to use domain from env

* Updated Sample env

* Fixed issues with minio deployment

* Opened messages and health_check APIs

* M8F-89: [Feature] Fixed PR comments

* M8F-89: [Feature] Fixed the docker compose builds

* Conflicts resolved

* AWS Deployment fixed

* Added Guards to keycloak_service.py and cookie_path_patch.py

* Moved out changes in spiffworkflow-frontend/bin/boot_server_in_docker

* Commented realm import

* Fixed Issues with Roles, Nats container

* keycloak-entrypoint.sh

Updated defaults for SUPERADMIN_USER and SUPERADMIN_PASS

* Fix line endings in entrypoint script for compatibility.Updated backend configuration in sample.env to change ports and URLs for local development.

* PR comments

* Fixed Routing issues for keycloak

* M8F-159: [Feature] Uncommented necessary lines from sample.env

* M8F-159: [Feature] Updated sample.env to use localhost instead of LOCAL_IP

* M8F-159: [Feature] Made changes According to PR comments

-Changed /health to /ping
-Updated documentation and comments

* M8F-159: [Feature] Moved comments form sample.env to relevant docs

* M8F-159: [Feature] Removed unused line of code

---------

Co-authored-by: auslin-aot <99173163+auslin-aot@users.noreply.github.com>

Author: vinaayakh-aot

README.md

@@ -77,7 +77,7 @@ A sample environment file is provided at the repository root. Create a working e
 cp sample.env .env
 ```
 
-Edit the `.env` file if adjustments are required for the local setup. Key tenancy options (`M8FLOW_DEFAULT_TENANT_ID`, `M8FLOW_ALLOW_MISSING_TENANT_CONTEXT`, `M8FLOW_TENANT_CLAIM`) are described in `sample.env`.
+Edit the `.env` file if adjustments are required for the local setup. **Environment variable meanings and examples** are documented in [docs/env-reference.md](docs/env-reference.md) (canonical reference; avoid duplicating long env explanations in this README). Variable names for tenancy and other areas also appear in `sample.env`.
 
 ---
 
@@ -217,7 +217,7 @@ After the containers start, continue below to the Keycloak Setup to import the r
 
 ### Import Realm
 
-You can import realms manually as below, or run `./extensions/m8flow-backend/keycloak/start_keycloak.sh` after starting Docker to import the identity realm and tenant-a.
+You can import realms by running `./extensions/m8flow-backend/keycloak/start_keycloak.sh` after starting Docker to import the identity realm.
 
 In the Keycloak Admin Console http://localhost:7002/ log in using the configured administrator credentials.
 
@@ -232,6 +232,8 @@ Browse or copy the content of `extensions/m8flow-backend/keycloak/realm_exports/
     <img src="./docs/images/keycloak-realm-settings-2.png" />
 </div>
 
+For Keycloak-related URL and issuer settings in `.env`, see [docs/env-reference.md](docs/env-reference.md) (Keycloak URLs).
+
 ### Configure Client
 
 With the realm "spiffworkflow" selected, click on "Clients" and then on the client ID **spiffworkflow-backend**.

docker/DEPLOYMENT.md

@@ -37,7 +37,7 @@ export TAG="latest"
 
 Ensure the image is built from the repo that includes the m8flow-backend extension (tenancy, Keycloak realm APIs, and support for `SPIFFWORKFLOW_BACKEND_WSGI_PATH_PREFIX` and `M8FLOW_KEYCLOAK_ADMIN_PASSWORD`). ECS expects the API under `/api/v1.0/*`.
 
-The backend uses the Keycloak **superadmin** user by default (username `superadmin`, created by the Keycloak image entrypoint). Set `KEYCLOAK_ADMIN_PASSWORD` or `M8FLOW_KEYCLOAK_ADMIN_PASSWORD` to the superadmin password (same as `KEYCLOAK_SUPERADMIN_PASSWORD` when using the Keycloak image entrypoint) so the backend can create realms and run partial import.
+The backend uses the Keycloak **super-admin** user by default (username `super-admin`, created by the Keycloak image entrypoint). Set `KEYCLOAK_ADMIN_PASSWORD` or `M8FLOW_KEYCLOAK_ADMIN_PASSWORD` to the super-admin password (same as `KEYCLOAK_SUPERADMIN_PASSWORD` when using the Keycloak image entrypoint) so the backend can create realms and run partial import.
 
 Build the production backend image (target `prod`, `linux/amd64` for ECS):
 

docker/README.md

@@ -2,6 +2,8 @@
 
 This directory contains the Docker setup for running M8Flow: Compose files, Dockerfiles for app services, and the Keycloak reverse-proxy config.
 
+**Environment variables:** Full meanings and examples live in [docs/env-reference.md](../docs/env-reference.md). This README only adds Docker Compose–specific behavior; do not duplicate the env reference here.
+
 ---
 
 ## File reference
@@ -100,6 +102,8 @@ Use with: `docker compose -f docker/m8flow-docker-compose.yml -f docker/m8flow-d
 
 Set production values in `.env` (e.g. `KEYCLOAK_HOSTNAME`, `M8FLOW_BACKEND_DATABASE_URI`, secrets) before running.
 
+**Docker Compose caveat:** The `m8flow-backend` service sets `KEYCLOAK_URL` and `M8FLOW_KEYCLOAK_URL` to `http://keycloak-proxy:7002` so server-side calls use the proxy, while browsers use the public URL (often `http://localhost:7002` or `http://<host>:7002`). For all other env semantics, see [docs/env-reference.md](../docs/env-reference.md).
+
 ---
 
 ## Non-root and ports

docs/env-reference.md

@@ -0,0 +1,29 @@
+# Environment variable reference
+
+This file is the **canonical** place for environment variable meanings and examples. The root [README.md](../README.md) and [docker/README.md](../docker/README.md) link here instead of repeating full definitions, to reduce drift.
+
+## Keycloak URLs
+
+- `KEYCLOAK_HOSTNAME`: Browser/public base URL used to reach Keycloak (for example `http://localhost:7002`). If clients access from another machine, use `http://<host>:7002` (or your real hostname and port).
+- `KEYCLOAK_HOSTNAME_URL`: Public Keycloak base URL Keycloak uses for token issuer (`iss`). In this repo’s Docker Compose, `KC_HOSTNAME_URL` is wired from `KEYCLOAK_HOSTNAME`; set `KEYCLOAK_HOSTNAME` consistently with how users reach Keycloak.
+- `KEYCLOAK_HOSTNAME_HOST` (optional): Hostname segment passed to Keycloak as `KC_HOSTNAME` in [docker/m8flow-docker-compose.yml](../docker/m8flow-docker-compose.yml) (default `localhost`). Adjust if your deployment needs a different hostname for Keycloak’s own hostname configuration.
+- `KEYCLOAK_URL` / `M8FLOW_KEYCLOAK_URL`: Backend URL for Keycloak Admin/API calls. **Docker Compose:** set by compose to `http://keycloak-proxy:7002` for `m8flow-backend` (internal network). **Local dev:** often `http://localhost:7002` to match the proxy port on the host.
+- `M8FLOW_APP_PUBLIC_BASE_URL` (optional): Set when the app and Keycloak are exposed on different public hosts. If unset, `KEYCLOAK_HOSTNAME` is used for generated app-facing URLs where applicable.
+
+## Connector attachment paths
+
+For SMTP and Slack connectors:
+
+- `*_ATTACHMENTS_DIR`: Host/source path where files are read from.
+- `*_ATTACHMENTS_USER_ACCESS_DIR`: User-visible mounted path used in service-task file selection.
+
+Examples:
+
+- `M8FLOW_CONNECTOR_SMTP_ATTACHMENTS_DIR=../email_attachments`
+- `M8FLOW_CONNECTOR_SMTP_ATTACHMENTS_USER_ACCESS_DIR=/email_attachments`
+- `M8FLOW_CONNECTOR_SLACK_ATTACHMENTS_DIR=../slack_attachments`
+- `M8FLOW_CONNECTOR_SLACK_ATTACHMENTS_USER_ACCESS_DIR=/slack_attachments`
+
+## Advanced Keycloak auth configs
+
+For `SPIFFWORKFLOW_BACKEND_AUTH_CONFIGS` patterns (master realm, `admin-cli`, role mapping), see [extensions/m8flow-backend/keycloak/KEYCLOAK_SETUP.md](../extensions/m8flow-backend/keycloak/KEYCLOAK_SETUP.md).

extensions/m8flow-backend/bin/run_m8flow_backend.sh

@@ -38,7 +38,6 @@ fi
 
 export SPIFFWORKFLOW_BACKEND_DATABASE_URI="${M8FLOW_BACKEND_DATABASE_URI}"
 export SPIFFWORKFLOW_BACKEND_BPMN_SPEC_ABSOLUTE_DIR="${M8FLOW_BACKEND_BPMN_SPEC_ABSOLUTE_DIR}"
-export SPIFFWORKFLOW_BACKEND_RUN_DATA_SETUP="${SPIFFWORKFLOW_BACKEND_RUN_DATA_SETUP:-false}"
 
 if [[ "${M8FLOW_BACKEND_UPGRADE_DB:-true}" != "false" ]]; then
   cd "$repo_root/spiffworkflow-backend"

extensions/m8flow-backend/src/m8flow_backend/api.yml

@@ -3,7 +3,7 @@ info:
   version: 1.0.0
   title: m8flow-backend-extension
 paths:
-  /health:
+  /ping:
     get:
       summary: Health check for load balancers and monitoring
       description: Returns 200 when the process is up. Public, no authentication required.

extensions/m8flow-backend/src/m8flow_backend/config.py

@@ -30,8 +30,8 @@ def keycloak_public_issuer_base() -> str:
 
 
 def keycloak_admin_user() -> str:
-    """Master realm admin username (default: super-admin, created by Keycloak entrypoint)."""
-    return _get("KEYCLOAK_ADMIN_USER") or _get("M8FLOW_KEYCLOAK_ADMIN_USER") or "super-admin"
+    """Master realm admin username (default is created by Keycloak entrypoint)."""
+    return _get("KEYCLOAK_ADMIN_USER") or _get("M8FLOW_KEYCLOAK_ADMIN_USER")
 
 
 def keycloak_admin_password() -> str:

extensions/m8flow-backend/src/m8flow_backend/tenancy.py

@@ -28,7 +28,7 @@
 TENANT_CONTEXT_EXEMPT_PATH_PREFIXES: tuple[str, ...] = (
     "/.well-known",
     "/favicon.ico",
-    "/v1.0/health",
+    "/v1.0/ping",
     "/v1.0/healthy",
     "/v1.0/status",
     "/v1.0/openapi.json",

extensions/m8flow-backend/src/m8flow_backend/utils/openapi_merge.py

@@ -35,7 +35,7 @@ def unified_add_api(self, specification, **kwargs):
 
                 # 3. Define extension path prefix and root-level paths (no prefix)
                 ext_prefix = "/m8flow"
-                ROOT_LEVEL_EXTENSION_PATHS = ("/health",)
+                ROOT_LEVEL_EXTENSION_PATHS = ("/ping",)
 
                 # 4. Merge Paths - keep core paths unchanged, add extension paths with prefix
                 # Core paths remain as-is; root-level extension paths stay as-is, others get /m8flow prefix

sample.env

@@ -39,9 +39,9 @@ M8FLOW_BACKEND_BPMN_SPEC_ABSOLUTE_DIR=./process_models
 M8FLOW_TEMPLATES_STORAGE_DIR=./templates
 
 # --- Auth (OIDC / Keycloak) ---
-# KEYCLOAK_HOSTNAME=http://localhost:7002  # URL used to reach Keycloak; set to http://<host>:7002 when accessing from another machine
-# KEYCLOAK_HOSTNAME_URL=http://localhost:7002  # Public Keycloak base URL; tokens use this as issuer (iss)
-# With Docker Compose, KEYCLOAK_URL is set to http://keycloak-proxy:7002 so the backend uses the proxy.
+# KEYCLOAK_HOSTNAME=http://localhost:7002
+# KEYCLOAK_HOSTNAME_URL=http://localhost:7002
+# See docs: README Keycloak Setup (hostname/issuer notes).
 M8FLOW_KEYCLOAK_URL=http://localhost:7002
 M8FLOW_BACKEND_OPEN_ID_SERVER_URL=http://localhost:7002/realms/spiffworkflow
 M8FLOW_BACKEND_OPEN_ID_CLIENT_ID=spiffworkflow-backend
@@ -54,10 +54,8 @@ M8FLOW_KEYCLOAK_SPOKE_KEYSTORE_P12=extensions/m8flow-backend/keystore.p12
 # Keystore password; required for tenant login flow when using spoke keystore.
 M8FLOW_KEYCLOAK_SPOKE_KEYSTORE_PASSWORD=
 
-# Optional: SPIFFWORKFLOW_BACKEND_AUTH_CONFIGS for Keycloak (identifier should match frontend cookie).
-# For master-realm super-admin browser access, add a second config: identifier=master, uri=.../realms/master.
-# If you also use API-style admin-cli tokens, add additional_valid_client_ids=admin-cli; the backend can map
-# matching M8Flow realm roles from realm_access.roles when no groups claim is present.
+# Optional Keycloak auth configs (identifier should match frontend cookie).
+# Advanced master/admin-cli config notes: extensions/m8flow-backend/keycloak/KEYCLOAK_SETUP.md
 SPIFFWORKFLOW_BACKEND_AUTH_CONFIGS__0__identifier=spiffworkflow-local
 SPIFFWORKFLOW_BACKEND_AUTH_CONFIGS__0__label=Keycloak
 SPIFFWORKFLOW_BACKEND_AUTH_CONFIGS__0__uri=http://localhost:7002/realms/spiffworkflow-local
@@ -69,7 +67,7 @@ SPIFFWORKFLOW_BACKEND_AUTH_CONFIGS__1__client_id=spiffworkflow-backend
 SPIFFWORKFLOW_BACKEND_AUTH_CONFIGS__1__client_secret=JXeQExm0JhQPLumgHtIIqf52bDalHz0q
 SPIFFWORKFLOW_BACKEND_AUTH_CONFIGS__1__additional_valid_client_ids=admin-cli
 SPIFFWORKFLOW_BACKEND_OPEN_ID_ADDITIONAL_VALID_ISSUERS=http://localhost:7002/realms/master,http://localhost:7002/realms/m8flow
-# Optional local-development bootstrap for the master realm browser client and global admin user.
+# Optional local bootstrap for master realm browser client and global admin user.
 # M8FLOW_KEYCLOAK_MASTER_CLIENT_SECRET=JXeQExm0JhQPLumgHtIIqf52bDalHz0q
 # KEYCLOAK_SUPER_ADMIN_USER=super-admin
 # KEYCLOAK_SUPER_ADMIN_PASSWORD=super-admin
@@ -78,7 +76,7 @@ SPIFFWORKFLOW_BACKEND_OPEN_ID_ADDITIONAL_VALID_ISSUERS=http://localhost:7002/rea
 M8FLOW_BACKEND_CELERY_ENABLED=true
 M8FLOW_BACKEND_CELERY_BROKER_URL=redis://localhost:6379/0
 M8FLOW_BACKEND_CELERY_RESULT_BACKEND=redis://localhost:6379/0
-# Backend/API should be false; the worker/flower launchers override this inside their own containers.
+# Keep false for backend/API; worker and flower launchers override.
 # M8FLOW_BACKEND_RUNNING_IN_CELERY_WORKER=false
 M8FLOW_BACKEND_RUN_BACKGROUND_SCHEDULER_IN_CREATE_APP=true
 M8FLOW_BACKEND_CELERY_ENABLE_EVENTS=true
@@ -90,34 +88,34 @@ M8FLOW_BACKEND_CELERY_AUTOSCALE_MAX=4
 # M8FLOW_BACKEND_CELERY_CONCURRENCY=2
 
 # ---Flower ---
-# Optional basic auth for Flower, format user:password (empty disables auth).
+# Optional Flower basic auth: user:password (empty disables).
 M8FLOW_BACKEND_CELERY_FLOWER_BASIC_AUTH=admin:admin
 # Optional worker/flower mount overrides.
 # M8FLOW_BACKEND_CELERY_PROCESS_MODELS_MOUNT_SOURCE=process_models_cache
 
 
 # --- Connector proxy ---
-# CONNECTOR_PROXY_PORT=8004
-# M8FLOW_BACKEND_CONNECTOR_PROXY_URL=http://localhost:8004
+CONNECTOR_PROXY_PORT=8004
+M8FLOW_BACKEND_CONNECTOR_PROXY_URL=http://localhost:8004
 # --- M8Flow Connector: SMTP ---
-# Local directory for connector to map attachments to.
+# SMTP attachment source dir (host path).
 M8FLOW_CONNECTOR_SMTP_ATTACHMENTS_DIR=../email_attachments
-# Mapped volume name for attachments directory. Users will select files from this directory when attaching files in the service task. E.g.: "/email_attachments/file.pdf".
+# SMTP attachment user-visible mount path in service tasks.
 M8FLOW_CONNECTOR_SMTP_ATTACHMENTS_USER_ACCESS_DIR=/email_attachments
 M8FLOW_CONNECTOR_SMTP_ATTACHMENTS_LIMIT_IN_MB=10
 M8FLOW_CONNECTOR_SMTP_TIMEOUT_SECONDS=300
 
 # --- M8Flow Connector: Slack ---
-# Local directory for connector to map Slack attachments from (host path for sync).
+# Slack attachment source dir (host path).
 M8FLOW_CONNECTOR_SLACK_ATTACHMENTS_DIR=../slack_attachments
-# Mapped volume path for attachments directory. Users select files from this path when attaching in the service task. E.g.: "/slack_attachments/sample.txt".
+# Slack attachment user-visible mount path in service tasks.
 M8FLOW_CONNECTOR_SLACK_ATTACHMENTS_USER_ACCESS_DIR=/slack_attachments
 M8FLOW_CONNECTOR_SLACK_UPLOAD_FILE_LIMIT_MB=50
 # --- Permissions ---
 M8FLOW_BACKEND_PERMISSIONS_FILE_ABSOLUTE_PATH=./extensions/m8flow-backend/src/m8flow_backend/config/permissions/m8flow.yml
 #M8FLOW_BACKEND_PERMISSIONS_FILE_NAME=m8flow.yml
 
-# Multitenant: true = tenant selection at / and /tenants/check; false = single-tenant.
+# Multitenant mode: true enables tenant selection routes.
 MULTI_TENANT_ON=false
 
 # --- SQLAlchemy Logs ---
@@ -128,27 +126,27 @@ M8FLOW_BACKEND_ENCRYPTION_LIB=cryptography
 M8FLOW_BACKEND_ENCRYPTION_KEY=0123456789abcdef0123456789abcdef
 
 # --- Tenancy ---
-# Default tenant id; one Keycloak realm per tenant.
+# Default tenant id.
 M8FLOW_DEFAULT_TENANT_ID=default
 M8FLOW_ALLOW_MISSING_TENANT_CONTEXT=true
-# JWT claim name for tenant id. Default: m8flow_tenant_id
+# JWT claim name for tenant id.
 # M8FLOW_TENANT_CLAIM=m8flow_tenant_id
 
 # --- Keycloak ---
 KEYCLOAK_ADMIN=admin
-# Keycloak admin app: URL, realm, admin user. Backend accepts KEYCLOAK_URL (or M8FLOW_KEYCLOAK_URL).
-# Set KEYCLOAK_HOSTNAME to the URL users/browser use to reach Keycloak so the backend accepts the token iss claim (e.g. http://localhost:7002 or http://localhost:7002). Required when using Docker or a proxy.
+# Keycloak admin app settings.
+# See docs: README Keycloak Setup and docs/env-reference.md.
 KEYCLOAK_HOSTNAME=http://localhost:7002
-# When Keycloak and the app are on different hosts, set M8FLOW_APP_PUBLIC_BASE_URL (e.g. https://app.example.com or http://localhost:7001). When not set, KEYCLOAK_HOSTNAME is used.
+# Optional public app URL when app and Keycloak are on different hosts.
 # M8FLOW_APP_PUBLIC_BASE_URL=
 KEYCLOAK_URL=http://localhost:7002
 KEYCLOAK_REALM=tenant-a
 KEYCLOAK_ADMIN_USER=admin
-# Admin password for create-realm API (KEYCLOAK_ADMIN_PASSWORD or M8FLOW_KEYCLOAK_ADMIN_PASSWORD).
+# Admin password for create-realm API.
 KEYCLOAK_ADMIN_PASSWORD=admin
 KEYCLOAK_CLIENT_ID=my-backend-app
 KEYCLOAK_CLIENT_SECRET=
-# Optional mTLS: KEYCLOAK_CERT_FILE, KEYCLOAK_KEY_FILE.
+# Optional mTLS cert/key paths.
 # KEYCLOAK_CERT_FILE=
 # KEYCLOAK_KEY_FILE=
 # Keycloak DB (used by docker/m8flow-docker-compose.yml for keycloak-db and keycloak service)