Merge pull request #186 from LandRegistry/develop

3.2.0

Author: sichapman

.github/workflows/linter.yml

@@ -20,7 +20,7 @@ jobs:
         with:
           fetch-depth: 0
         # The only difference between full and slim is the latter excludes .NET and Rust linters
-      - uses: super-linter/super-linter/slim@v7
+      - uses: super-linter/super-linter/slim@v8
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           LINTER_RULES_PATH: .

README.md

@@ -131,6 +131,8 @@ The list of allowable commodity values is:
 16. ibmmq
 17. localstack
 18. valkey
+19. prometheus
+20. grafana
 
 The file may optionally also indicate that one or more services are resource intensive ("expensive") when starting up. The dev env will start those containers seperately - 3 at a time - and wait until each are declared healthy (or crash and get restarted 10 times) before starting any more.
 
@@ -224,11 +226,11 @@ There are no fragments needed when using this. The Management Console will be av
 
 Rabbit is available over port 5672 and TLS on port 5671.
 
-TLS presents a self signed cert. If verification is needed a copy of the ca certificate is [here](scripts/docker/rabbitmq/certs/ca_certificate.crt). The host has been set to `rabbitmq` for host verification in most common libraries, although will only work within the docker network.
+TLS presents a self signed cert. If verification is needed then use the provided [ca certificate](scripts/docker/rabbitmq/certs/ca_certificate.crt). The host has been set to `rabbitmq` for host verification in most common libraries, although will only work within the docker network.
 
 MTLS is not enabled, although a [client certificate pem](scripts/docker/rabbitmq/certs/client_certificate.pem) and [client key pem](scripts/docker/rabbitmq/certs/client_key.pem) have been generated as part of the certificate set for potential future use.
 
-Currently, only the `rabbitmq_management`, `rabbitmq_consistent_hash_exchange`, `rabbitmq_shovel`, `rabbitmq_shovel_management` and `rabbitmq_stream` plugins are enabled.
+Currently, only the `rabbitmq_management`, `rabbitmq_consistent_hash_exchange`, `rabbitmq_shovel`, `rabbitmq_shovel_management`, `rabbitmq_stream` and `rabbitmq_prometheus` plugins are enabled.
 
 ##### ActiveMQ
 
@@ -249,6 +251,15 @@ bashin redis
 redis-cli monitor
 ```
 
+##### Prometheus
+
+Prometheus will be available at <http://localhost:9090>. The scrape config lives in `dev-env-config/prometheus/prometheus.yml` and is mounted by any app that needs Prometheus.
+For production, avoid high-cardinality RabbitMQ metrics unless needed. Prefer to keep only specific queues with `metric_relabel_configs`, or disable per-queue metrics at the broker if you only need aggregate health.
+
+##### Grafana
+
+Grafana will be available at <http://localhost:3000> (admin/admin). Provisioning is defined in `dev-env-config/grafana/provisioning/`. Dashboards live inside each app at `apps/<app>/fragments/grafana/dashboards/` and are mounted into Grafana by the app’s compose fragment.
+
 ##### Squid
 
 There are no fragments needed when using this. An HTTP proxy will be made available to all containers at runtime, at hostname `squid` and port 3128. It will be available on the host on port 30128.
@@ -300,7 +311,7 @@ Applications using OAuth flows or the OpenID Connect protocol can use Keycloak f
 
 JWT tokens issued from the `development` realm have been configured to mimic those issued by Microsoft ADFS servers. In particular, the LDAP `cn` field is mapped to the `UserName` claim in JWT tokens along with the `Office` claim mapped from the `physicalDeliveryOfficeName` in the LDAP database and the `group` claim listing the user's group memberships.
 
-A [JSON export](scripts/docker/auth/keycloak/development_realm.json) of the `development` realm is used to configure the realm. If further configuration of the realm is required, you can make changes in the admin console and re-export the realm using the procedure described in "Exporting a realm" [here](https://hub.docker.com/r/jboss/keycloak/#exporting-a-realm).
+A [JSON export](scripts/docker/auth/keycloak/development_realm.json) of the `development` realm is used to configure the realm. If further configuration of the realm is required, you can make changes in the admin console and re-export the realm using the procedure described in "Exporting a realm" section of the [documentation](https://hub.docker.com/r/jboss/keycloak/#exporting-a-realm).
 
 The exported JSON can then be merged back into this repository and reused.
 
@@ -334,7 +345,7 @@ _Running Cadence web locally_
 
 A default Localstack configuration is provided with a minimal number of enabled services available (S3 only at present). Localstack does not _require_ the use of any other external configuration file (as applications can manage buckets programatically through methods such as the [AWS SDK](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3-buckets.html)).
 
-However, if additional configuration (such as new buckets) are necessary before application startup, you can use a `localstack-init-fragment.sh` to perform this provisioning; an example of which is provided [here](snippets/localstack-init-fragment.sh).
+However, if additional configuration (such as new buckets) are necessary before application startup, you can use a `localstack-init-fragment.sh` to perform this provisioning; like [this example](snippets/localstack-init-fragment.sh).
 
 Localstack is available at <http://localstack:4566> within the Docker network, and <http://localhost:4566> on the host.
 

scripts/add-aliases.sh

@@ -165,7 +165,8 @@ function fullreset(){
 
 function alembic(){
     ex -e SQL_USE_ALEMBIC_USER=yes -e SQL_PASSWORD=superroot -e SQLALCHEMY_POOL_RECYCLE=3600 ${1} \
-        bash -c 'cd /src && python3 manage.py db '"${@:2}"''
+        bash -c 'cd /opt/app-root/src && [ -d "$VENV_DIR" ] && source $VENV_DIR/bin/activate;
+        python3 manage.py db '"${@:2}"''
 }
 
 function devenv-help(){

scripts/docker/grafana/Dockerfile

@@ -0,0 +1 @@
+FROM grafana/grafana-oss:12.3.3

scripts/docker/grafana/compose-fragment.yml

@@ -0,0 +1,8 @@
+services:
+  grafana:
+    container_name: grafana
+    build: ../scripts/docker/grafana/
+    ports:
+      - 3000:3000
+    environment:
+      GF_SECURITY_ADMIN_PASSWORD: admin

scripts/docker/prometheus/Dockerfile

@@ -0,0 +1,2 @@
+FROM prom/prometheus:v3.9.1
+COPY prometheus.yml /etc/prometheus/prometheus.yml

scripts/docker/prometheus/compose-fragment.yml

@@ -0,0 +1,6 @@
+services:
+  prometheus:
+    container_name: prometheus
+    build: ../scripts/docker/prometheus/
+    ports:
+      - 9090:9090

scripts/docker/prometheus/prometheus.yml

@@ -0,0 +1,8 @@
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: prometheus
+    static_configs:
+      - targets: ["prometheus:9090"]

scripts/docker/rabbitmq/Dockerfile

@@ -16,4 +16,6 @@ RUN rabbitmq-plugins --offline enable rabbitmq_shovel rabbitmq_shovel_management
 # This is a RabbitMQ plugin that exposes streams. 
 # Streams are a new persistent and replicated data structure, handy to persist historic messages for a period of time
 # https://www.rabbitmq.com/stream.html
-RUN rabbitmq-plugins --offline enable rabbitmq_stream
+# Expose Prometheus metrics for monitoring/dashboards.
+RUN rabbitmq-plugins --offline enable rabbitmq_stream && \
+    rabbitmq-plugins --offline enable rabbitmq_prometheus

scripts/docker/rabbitmq/compose-fragment.yml

@@ -4,5 +4,12 @@ services:
     build: ../scripts/docker/rabbitmq/
     ports:
       - 15672:15672
+      - 15692:15692
       - 5672:5672
       - 5671:5671
+    # Lightweight TCP check on AMQP port (RabbitMQ recommend this over rabbitmq-diagnostics ping).
+    healthcheck:
+      test: ["CMD-SHELL", "bash -c '</dev/tcp/127.0.0.1/5672' || exit 1"]
+      interval: 10s
+      timeout: 5s
+      retries: 10