Merge pull request #188 from LandRegistry/develop

3.3.0

Author: sichapman

README.md

@@ -120,19 +120,21 @@ The list of allowable commodity values is:
 5. elasticsearch7
 6. nginx
 7. rabbitmq
-8. redis
-9. swagger
-10. wiremock
-11. squid
-12. auth
-13. cadence
-14. cadence-web
-15. activemq
-16. ibmmq
-17. localstack
-18. valkey
-19. prometheus
-20. grafana
+8. rabbitmq-4
+9. redis
+10. swagger
+11. wiremock
+12. squid
+13. auth
+14. cadence
+15. cadence-web
+16. activemq
+17. ibmmq
+18. localstack
+19. valkey
+20. prometheus
+21. grafana
+22. govuk-one-login
 
 The file may optionally also indicate that one or more services are resource intensive ("expensive") when starting up. The dev env will start those containers seperately - 3 at a time - and wait until each are declared healthy (or crash and get restarted 10 times) before starting any more.
 
@@ -220,18 +222,34 @@ This is a file that contains stub mappings that Wiremock will pick up and use, a
 
 This is a directory that can contain multiple stub mappings files that Wiremock will pick up and use. The files should all be in the same format as the example above. If such a wiremock directory exists, it will take precedence over any `wiremock-fragment.json` file.
 
-##### RabbitMQ
+##### RabbitMQ 3
 
 There are no fragments needed when using this. The Management Console will be available on <http://localhost:15672> (guest/guest).
 
-Rabbit is available over port 5672 and TLS on port 5671.
+RabbitMQ 3 is available over port 5672 and TLS on port 5671.
 
 TLS presents a self signed cert. If verification is needed then use the provided [ca certificate](scripts/docker/rabbitmq/certs/ca_certificate.crt). The host has been set to `rabbitmq` for host verification in most common libraries, although will only work within the docker network.
 
 MTLS is not enabled, although a [client certificate pem](scripts/docker/rabbitmq/certs/client_certificate.pem) and [client key pem](scripts/docker/rabbitmq/certs/client_key.pem) have been generated as part of the certificate set for potential future use.
 
 Currently, only the `rabbitmq_management`, `rabbitmq_consistent_hash_exchange`, `rabbitmq_shovel`, `rabbitmq_shovel_management`, `rabbitmq_stream` and `rabbitmq_prometheus` plugins are enabled.
 
+The default queue type for queues in RabbitMQ 3 is `classic`.
+
+##### RabbitMQ 4
+
+There are no fragments needed when using this. The Management Console will be available on <http://localhost:15682> (guest/guest).
+
+RabbitMQ 4 is available over port 5682 and TLS on port 5681.
+
+TLS presents a self signed cert. If verification is needed then use the provided [ca certificate](scripts/docker/rabbitmq/certs/ca_certificate.crt). The host has been set to `rabbitmq` for host verification in most common libraries, although will only work within the docker network.
+
+MTLS is not enabled, although a [client certificate pem](scripts/docker/rabbitmq/certs/client_certificate.pem) and [client key pem](scripts/docker/rabbitmq/certs/client_key.pem) have been generated as part of the certificate set for potential future use.
+
+Currently, only the `rabbitmq_management`, `rabbitmq_consistent_hash_exchange`, `rabbitmq_shovel`, `rabbitmq_shovel_management`, `rabbitmq_stream` and `rabbitmq_prometheus` plugins are enabled.
+
+The default queue type for queues in RabbitMQ 4 is `quorum`.
+
 ##### ActiveMQ
 
 There are no fragments needed when using this. The Management Console will be available on <http://localhost:8161> (admin/admin).
@@ -362,6 +380,16 @@ bashin redis
 redis-cli monitor
 ```
 
+###### GOV.UK One Login
+
+The GOV.UK One Login simulator is a development tool that lets you run end-to-end tests, so you can:
+
+- test and verify specific user information such as names and email addresses
+- request specific error codes and learn more about what they mean in a test environment
+- test integrations with GOV.UK One Login without going through account creation
+
+See [documentation](https://github.com/govuk-one-login/simulator).
+
 #### Other files
 
 **`/fragments/custom-provision.sh`**

logic.rb

@@ -75,7 +75,7 @@
 
 # Does a version check and self-update if required
 if options['self_update']
-  this_version = '3.2.0'
+  this_version = '3.3.0'
   puts colorize_lightblue("This is a universal dev env (version #{this_version})")
   # Skip version check if not on master (prevents infinite loops if you're in a branch that isn't up to date with the
   # latest release code yet)

scripts/docker/govuk-one-login/Dockerfile

@@ -0,0 +1 @@
+FROM ghcr.io/govuk-one-login/simulator:latest

scripts/docker/govuk-one-login/compose-fragment.yml

@@ -0,0 +1,6 @@
+services:
+  govuk-one-login:
+    container_name: govuk-one-login
+    build: ../scripts/docker/govuk-one-login/
+    ports:
+      - "3000:3000"

scripts/docker/logging/server.py

@@ -3,7 +3,6 @@
 import socketserver
 import threading
 
-
 HOST, PORT = "0.0.0.0", 25826
 
 syslog_pattern = re.compile(r"\S* (\S*) \S* (\S*) \d+ \S* (?:- )?(.*)")
@@ -23,7 +22,7 @@ class ThreadedTCPRequestHandler(socketserver.StreamRequestHandler):
     """
 
     def handle(self):
-        while(True):
+        while True:
             data = bytes.decode(self.rfile.readline().strip())
             if data == "":
                 print("Connection closed", flush=True)
@@ -34,8 +33,8 @@ def handle(self):
             if m is None:
                 # Not syslog format for some reason, just send raw message to console and file
                 print(data, flush=True)
-                #with open("/log-dir/log.txt", 'a') as file_:
-                    #file_.write(data + '\n')
+                # with open("/log-dir/log.txt", 'a') as file_:
+                # file_.write(data + '\n')
             else:
                 # Now, is the message valid JSON?
                 try:
@@ -48,14 +47,14 @@ def handle(self):
                     # Send nicely structured message to console and file
                     data = "{0} {1} {2} {3} {4} {5}".format(m.group(1), m.group(2), level, message, exception, traceid)
                     print(data, flush=True)
-                    #with open("/log-dir/log.txt", 'a') as file_:
-                        #file_.write(data + '\n')
+                    # with open("/log-dir/log.txt", 'a') as file_:
+                    # file_.write(data + '\n')
                 except ValueError:
                     # Nope. Send half-structured message to console and file
                     data = "{0} {1} {2}".format(m.group(1), m.group(2), m.group(3))
                     print(data, flush=True)
-                    #with open("/log-dir/log.txt", 'a') as file_:
-                        #file_.write(data + '\n')
+                    # with open("/log-dir/log.txt", 'a') as file_:
+                    # file_.write(data + '\n')
 
 
 if __name__ == "__main__":

scripts/docker/rabbitmq-4/Dockerfile

@@ -0,0 +1,20 @@
+# This is so a reset (dc down) won't remove the base rabbit image, only the one created from this dockerfile
+FROM docker.io/rabbitmq:4.2-management-alpine
+# Copy the configuration file to rabbitmq
+# https://www.rabbitmq.com/configure.html#configuration-files
+COPY rabbitmq.conf /etc/rabbitmq/
+# Copy certs for TLS
+# Nice tool here from rabbit for generating https://github.com/rabbitmq/tls-gen
+COPY certs/ca_certificate.pem certs/server_certificate.pem certs/server_key.pem certs/
+# This plugin adds a consistent-hash exchange type to RabbitMQ. This exchange type uses consistent hashing to 
+# distribute messages between the bound queues
+RUN rabbitmq-plugins --offline enable rabbitmq_consistent_hash_exchange
+# This is a RabbitMQ plugin adds message transfer and management functionality.
+# Handy for tasks such as bulk moving msgs for a DLQ.
+# https://www.rabbitmq.com/shovel.html
+RUN rabbitmq-plugins --offline enable rabbitmq_shovel rabbitmq_shovel_management
+# This is a RabbitMQ plugin that exposes streams. 
+# Streams are a new persistent and replicated data structure, handy to persist historic messages for a period of time
+# https://www.rabbitmq.com/stream.html
+RUN rabbitmq-plugins --offline enable rabbitmq_stream && \
+    rabbitmq-plugins --offline enable rabbitmq_prometheus

scripts/docker/rabbitmq-4/certs/ca_certificate.crt

@@ -0,0 +1,21 @@
+-----BEGIN CERTIFICATE-----
+MIIDezCCAmOgAwIBAgIJALM67Mx3qs5qMA0GCSqGSIb3DQEBCwUAMEwxOzA5BgNV
+BAMMMlRMU0dlblNlbGZTaWduZWR0Um9vdENBIDIwMjItMDctMjJUMTc6NTc6MjMu
+Njc3MTQwMQ0wCwYDVQQHDAQkJCQkMB4XDTIyMDcyMjE2NTcyM1oXDTMyMDcxOTE2
+NTcyM1owTDE7MDkGA1UEAwwyVExTR2VuU2VsZlNpZ25lZHRSb290Q0EgMjAyMi0w
+Ny0yMlQxNzo1NzoyMy42NzcxNDAxDTALBgNVBAcMBCQkJCQwggEiMA0GCSqGSIb3
+DQEBAQUAA4IBDwAwggEKAoIBAQDfb+RXOj3kCE397cnE665FGJr3RxTrcUdE08Fu
+5W8ocwpXQzo2j0qyKrJ09U6ASMnBhhVkALv6qGWkjNLcAVDv+kxLPV2pg7yrT+xz
+IklhtsUkkko9GHV++2mWvbW1cFWZR38trUjrEVPO3zoclJTwAO6B4u48ZhFocxie
+Gpg28fHVnKf/Z6ziMAu54kLRbCNM2F3/XUl0l/A4l4uyCRVaEZRq/v+TaMYs4hLx
+6TgsAZ43daVO9xG/qQRiD+Sjpgj5e85s6THTct6+ahU2Rkwa1wYlGbSCIq8QaGop
+1R7+J9PLwWmiQWriHUHccegNGhwmcv456gFgB8EldgYBpUOXAgMBAAGjYDBeMA8G
+A1UdEwEB/wQFMAMBAf8wCwYDVR0PBAQDAgEGMB0GA1UdDgQWBBQAMB5YaNLrhwjn
+NPLo8xefe+eF1DAfBgNVHSMEGDAWgBQAMB5YaNLrhwjnNPLo8xefe+eF1DANBgkq
+hkiG9w0BAQsFAAOCAQEANk0Yn2lV+9jwYEhGq7HlFoluf9KnZhn2U1/WGQjUVu6C
+MLBQ3L3ikZef9UacnSDt0DdlpKorn8ujnc9xcOJKj3TqkKqIIHabagAPFQUV5yig
+IYgGXZWxNJGia/ho4KG43F7HOh6MaZyHAT9fWsUKhzXeUDFHXGHjV4WFfyX3K/XH
+KIvfLrMPd2UIt9udoA0unprRbBRLCRHmV8RUYvIqCT+VWH29GWz4CpvpZqlDP7uC
+3zaLzb88jQg9zjCh7sDFt+em/1dRRJ54PpgddFtf2qVTYBJrVgTrqey+nzewnAAd
+fTb+ZUIjQNlpoek8gz/5oQtNSDAuZVa52/VAFXh1/g==
+-----END CERTIFICATE-----

scripts/docker/rabbitmq-4/certs/ca_certificate.pem

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDfb+RXOj3kCE39
+7cnE665FGJr3RxTrcUdE08Fu5W8ocwpXQzo2j0qyKrJ09U6ASMnBhhVkALv6qGWk
+jNLcAVDv+kxLPV2pg7yrT+xzIklhtsUkkko9GHV++2mWvbW1cFWZR38trUjrEVPO
+3zoclJTwAO6B4u48ZhFocxieGpg28fHVnKf/Z6ziMAu54kLRbCNM2F3/XUl0l/A4
+l4uyCRVaEZRq/v+TaMYs4hLx6TgsAZ43daVO9xG/qQRiD+Sjpgj5e85s6THTct6+
+ahU2Rkwa1wYlGbSCIq8QaGop1R7+J9PLwWmiQWriHUHccegNGhwmcv456gFgB8El
+dgYBpUOXAgMBAAECggEAAxCi3jtzS+R9RWV0phnhixsaTRgRhmVIZ/5LofzNuI1T
+b/0BGGyAxs7lIcuXpxTYh6oSMwv7iCW1NzujTNdLx+70Cvgth7zB4su236FxxkBe
+lfDXydzDB9nlCedSTcnbsr2TVgoRfQAA2U6gmnHDVUN+AhQd1oBD7t3dUZXlzqMe
+bznn/CzHXonhHbHT79KbZHgb1lJGZQapjUGdwmAdjLt+flJcD7SCYabG/4tfwo/c
+L7EKeJb3Pye3mdVvs6sLMyPmrTxaT5PILllFwbzmcuoTmWi3AZ4e6qwygpGvtXLo
+oOlZfuShdXz2MCoG7KUEX/kgwHZy5jUivxyn7aqOAQKBgQD7kqEr87WJAhJ6TgFL
+T4ndmTYYuExEfIAu4D9UFuEiFH04GSMcT5RUpyZc20rIaL4sz8FVHYffqXbK24pl
+ScD2wtsb6sm9+tadtNxnihd2z5nT0z4SYYu13ImOWGeEri2UDXu0vnwCA9ULCq5w
+Sj9gyYxlZykTElW4C2gjhSlClwKBgQDjXoHU3jONJYa+QAqlmIKIB37fftkAnKsY
+yjN6hvlFU4U0OI6bxYSvjfIzSp/GEEHOXWNlFfkRBR7uMRUmMPYxxPq73c61pVO1
+d6BxTOK/tHNwi0pJeNRUYonyqlwnOwOfwkPh0BYudV7NbArq5eWMGGo8G5hkStZ/
+feQDbUEnAQKBgQDHXgeuTkAiaOr91A3AgjMu03mEk6GVcNkgzQRC/rLCEFy/9WPW
+ztYtVB3Z3svgEOoNl0OO0Y1ZWfJThzKhXJa4t+lusV3Z2gz1BFzbDRvVQ2aT8Uey
+x20x3amBjlvTUTm7qAfOUFr0Odc/HeLqCF07td42ik+J1rDp3thTtfbh1wKBgFnT
+6K9U6kEFZmqbmpk527gqXfGmh806GE6+8/+ge7LsAGBVR+I68uT4y4uAHAeAPL9v
+Hem9+JzEzSeFtLlV9q5LCFRYmWYiXyP+kyCwI5VhkYoRPdQxLJH6LE33CFlnc1MW
+yQLZ1ymbrUyzNsz+7uWeb7o3nMli18Oe1QqVi3wBAoGBAOIMYwFP5aYXktN9rYS+
+XQdKWy1PbetdAsEklf84J351A2Ggh+X7rE/eBvOySLEqncsaWXytZN0HBE6m5ztN
+XFe/N/jU3m4buefmFukznnTkhuu70lS/RBqWy+2IgRwO5l6sUrbst4VypJCi1xTD
+ExD4m0ADahDs3x+k60BL7x82
+-----END PRIVATE KEY-----

scripts/docker/rabbitmq-4/certs/ca_key.pem

@@ -0,0 +1,21 @@
+-----BEGIN CERTIFICATE-----
+MIIDhzCCAm+gAwIBAgIBAjANBgkqhkiG9w0BAQsFADBMMTswOQYDVQQDDDJUTFNH
+ZW5TZWxmU2lnbmVkdFJvb3RDQSAyMDIyLTA3LTIyVDE3OjU3OjIzLjY3NzE0MDEN
+MAsGA1UEBwwEJCQkJDAeFw0yMjA3MjIxNjU3MjRaFw0zMjA3MTkxNjU3MjRaMCQx
+ETAPBgNVBAMMCHJhYmJpdG1xMQ8wDQYDVQQKDAZjbGllbnQwggEiMA0GCSqGSIb3
+DQEBAQUAA4IBDwAwggEKAoIBAQDItDPr/y3UgpGrHOce2DtSVXHfWaoQ0NLrff1y
+RUAGKl1wR2C6WNeyXxA3e8Nee22aRE7S78A99uj6PscnbuboMvSopaTXBF6tFjCL
+eql+01YrzSYGMRhomse8Nm7INzMMO9g6JH8Uer0ddiIa+DAS3o6+6N507oXJUAvV
+jwTn91OkIidWLbR7Ma7tcrLQ8lpHSliNqXNut26QJ04hHgXlZRWapajqbhxwxvBr
++FuVw0GOyKUeYCnnY++lZGwZ9UP9qjcd9lb15jEOkHHqyJVOqByKrJbbKd2c3ZQD
+Sn1vkj8idXn4yvQrRxEaF7mqce2FxprHfPWPT5321COr4hpnAgMBAAGjgZswgZgw
+CQYDVR0TBAIwADALBgNVHQ8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwIwNwYD
+VR0RBDAwLoIIcmFiYml0bXGCF0FuZHlzLU1hY0Jvb2stUHJvLmxvY2Fsgglsb2Nh
+bGhvc3QwMAYDVR0fBCkwJzAloCOgIYYfaHR0cDovL2xvY2FsaG9zdDo4MDAwL2Jh
+c2ljLmNybDANBgkqhkiG9w0BAQsFAAOCAQEAXH4BlV00gvdPrAdp2UuF0G6VAKgr
+KesZ0lsVBx5jCNskeXkI1SU3BvfltrfY2PKaWVA0Sow+4jiZkLnFkC581AmlvEPg
+SgW0S/STdWojWrt3apBloIfapE++jQzcs3ATnaDeOQvb/fW/MUPZEuZfFn7wiX59
+c0obO1rqix5Qyf1JXyy38oh/4SdqeEXsI3l26qSmE7x41YbXv0d5wp+p6dXrut5n
+go/cC4TszSHb+/b6Nh+fFiPojqMDEpt5biTyZhGkJMIFrO9MvHnK6w9d9RUYRv3x
+aZ94ouB/16wit5ZUMGkFNPlHwwxKC7Aqare6PNzVi6RsIf4tZtgIzSvokg==
+-----END CERTIFICATE-----