Docker compose files for 3rd party, save&restore, alarm server

Author: georgweiss

services/README.md

@@ -0,0 +1,30 @@
+# Services
+
+Modules in this directory include CS Studio/Phoebus middleware services:
+
+* Alarm Config Logger
+* Alarm Logger
+* Alarm Service
+* Archive Engine
+* Save-and-Restore 
+
+Additional middleware services are maintained in other repositories:
+* Phoebus Logbook Service (Olog): https://github.com/Olog/phoebus-olog
+* Channel Finder Service: https://github.com/ChannelFinder/ChannelFinderService
+* Archiver Appliance: https://github.com/archiver-appliance/epicsarchiverap
+
+# Docker
+
+The provided Docker compose file can be used to launch the full stack of 3rd party services upon which
+services in this repository depend:
+
+* Elasticsearch
+  * Used by Save-and-Restore, Alarm Logger, Channel Finder and Olog
+* MongoDB 
+  * Used by Olog
+* Kafka + Zookeeper
+  * Used by Alarm Server and Alarm Logger
+
+The Docker compose file depends on the environment variable ```HOST_IP_ADDRESS```, which must be set
+to the IP address of the host running the Docker container. Kafka clients must use this as the
+```bootstrap-server``` IP address.

services/alarm-server/README.md

@@ -50,61 +50,13 @@ some useful startup arguments include
 
 ## Docker
 
-Docker compose files are provided for convenience.
+Docker compose files are provided for convenience to cover two use cases:
 
-### Launch only Kafka
+1. ```docker-compose-alarm-server-only.yml``` will run the alarm server. The 
+    environment variable ```KAFKA_HOST``` must be set to identify the Kafka server IP address,
+    while the environment variable ```CONFIG``` identifies the configuration (topic) name.
+2. ```docker-compose-alarm-server-only-import.yml``` will run the alarm server for the
+    purpose of importing a configuration. Environment variables must be se as in case 1.
+    Additionally, the environment variable ```CONFIG_FILE``` must identify an alarm
+    configuration file in the current user's home directory.
 
-``docker-compose-kafka.yml`` 
-
-This is used to start the Kafka message broker (in Zookeeper mode). Typical use case: launch alarm
-server manually.
-
-Usage:
-
-```
->docker compose -f docker-compose-kafka.yml --env-file /path/to/environment/file
-```
-
-where ``/path/to/environment/file`` is a file defining a ``HOSTNAME`` variable, e.g.
-```
-HOSTNAME=foo.bar.com
-```
-When launching the alarm server, the ``-server`` argument must match the value of ``HOSTNAME``.
-
-### Launch Kafka and alarm server for import
-
-``docker-compose-alarm-server-for-import.yml``
-
-This is used to start the full stack for the purpose of importing an alarm configuration file.
-
-Usage:
-
-```
->docker compose -f docker-compose-alarm-server-for-import.yml --env-file /path/to/environment/file
-```
-
-Here the ``/path/to/environment/file`` file *must* list:
-```
-HOSTNAME=foo.bar.com
-CONFIG_FILE=/path/to/Accelerator.xml
-```
-In other words, this imports an alarm configuration for the default topic ``Accelerator``.
-
-### Launch Kafka and alarm server
-
-``docker-compose-alarm-server-for-import.yml``
-
-This is used to start the full stack for running the alarm server and listen to EPICS alarms.
-
-Usage:
-
-```
->docker compose -f docker-compose-alarm-server.yml --env-file /path/to/environment/file
-```
-
-Here the ``/path/to/environment/file`` file *must* list:
-```
-HOSTNAME=foo.bar.com
-```
-
-Note that in all cases ``HOSTNAME`` *can not* be set to ``localhost``.

services/alarm-server/docker-compose-alarm-server-for-import.yml

@@ -0,0 +1,9 @@
+services:
+  alarmserver:
+    image: ghcr.io/controlsystemstudio/phoebus/service-alarm-server:csstudio-2989
+    volumes:
+      - /home:/home
+    command: >
+      /bin/bash -c "
+      java -jar /alarmserver/service-alarm-server-*.jar -config ${CONFIG} -import ${CONFIG_FILE} -server ${KAFKA_HOST}:9092"
+  

services/alarm-server/docker-compose-alarm-server-only-import.yml

@@ -1,14 +1,9 @@
 services:
   alarmserver:
-    image: ghcr.io/controlsystemstudio/phoebus/service-alarm-server:CSSTUDIO-2989
-    networks:
-      - middleware-net
+    image: ghcr.io/controlsystemstudio/phoebus/service-alarm-server:csstudio-2989
     volumes:
       - /home:/home
     command: >
       /bin/bash -c "
-      java -jar /alarmserver/service-alarm-server-*.jar -server kafka:9092"
-
-networks:
-  middleware-net:
-    driver: bridge
+      java -jar /alarmserver/service-alarm-server-*.jar -config ${CONFIG} -server ${KAFKA_HOST}:9092 -noshell"
+  

services/alarm-server/docker-compose-alarm-server-only.yml

@@ -2,8 +2,6 @@ services:
   elasticsearch:
     image: docker.elastic.co/elasticsearch/elasticsearch:8.11.2
     hostname: elastic
-    networks:
-      - middleware-net
     ports:
       - "9200:9200"
     environment:
@@ -15,8 +13,6 @@ services:
 
   mongo:
     image: mongo
-    networks:
-      - middleware-net
     ports:
       - 27017:27017
     volumes:
@@ -26,8 +22,6 @@ services:
   zookeeper:
     image: confluentinc/cp-zookeeper:latest
     hostname: zookeeper
-    networks:
-      - middleware-net
     healthcheck:
       test: nc -z localhost 2181 || exit -1
       interval: 10s
@@ -42,31 +36,28 @@ services:
   kafka:
     image: confluentinc/cp-kafka:latest
     hostname: kafka
-    networks:
-      - middleware-net
-    ports:
-      - 9092:9092
     depends_on:
       zookeeper:
         condition: service_healthy
+    ports:
+      - "9092:9092"
+      - "29092:29092"
     environment:
+      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:19092,EXTERNAL://${HOST_IP_ADDRESS}:9092,DOCKER://localhost:29092
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
+      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
+      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
       KAFKA_BROKER_ID: 1
-      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
-      KAFKA_LISTENERS: INTERNAL://0.0.0.0:9094,OUTSIDE://0.0.0.0:9092
-      KAFKA_ADVERTISED_LISTENERS: INTERNAL://:9094,OUTSIDE://:9092
-      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
       KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
-      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
-
+      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
+      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
 
 volumes:
   es-data:
     driver: local
   mongodata:
     driver: local
 
-networks:
-  middleware-net:
-    driver: bridge
 
 

services/alarm-server/docker-compose-alarm-server.yml

@@ -36,7 +36,7 @@ or you can use a `.properties file`
 To check that the server is running correctly.
 
 ```
-$ curl --fail-with-body http://localhost:8080/save-restore
+$ curl --fail-with-body http://localhost:8080/save-restore/
 {
   "name" : "service-save-and-restore",
   "version" : "4.7.4-SNAPSHOT",
@@ -58,6 +58,16 @@ Details about the REST API is found in the Phoebus bundled help content, which i
 
 # Docker
 
-The latest version of the service is available as a Docker image (ghcr.io/controlsystemstudio/phoebus/service-save-and-restore:master). Any pushes to the master branch into this directory will trigger a new build of the image.
+The latest version of the service is available as a Docker image (ghcr.io/controlsystemstudio/phoebus/service-save-and-restore:master). Pushes to the master branch into this directory will trigger a new build of the image.
 
-To run the service and its dependency (Elasticsearch), use docker-compose.yml.
+Docker compose files are provided. These depend on the environment variables as described below.
+
+1. ```docker-compose.yml```. Use this to launch both Elasticsearch and the service. The environment variable ```HOST_EXTERNAL_IP_ADDRESS``` 
+    should be set to the host's external IP address (i.e. **not** 127.0.0.1 or localhost).
+2. ```docker-compose-save-and-restore.yml```. Use this to launch save-and-restore service only. The environment variable ```HOST_EXTERNAL_IP_ADDRESS```
+   should be set to the host's external IP address (i.e. **not** 127.0.0.1 or localhost), while the environment variable ```ELASTIC_HOST``` should
+   be set to the IP address of the Elasticsearch host. If Elasticsearch is running on  localhost, please specify the host's
+   external IP address. 
+
+**NOTE:** Accessing IOCs over pva (default mode in the Docker compose files) works **only** if IOC is running on the
+same host as the Docker container. Moreover, this has been verified to work only on Linux.

services/alarm-server/docker-compose-kafka.yml

@@ -0,0 +1,20 @@
+services:
+  saveandrestore:
+    image: ghcr.io/controlsystemstudio/phoebus/service-save-and-restore:master
+    ports:
+      - "8080:8080"
+    environment:
+      ELASTICSEARCH_NETWORK_HOST: ${ELASTIC_HOST_IP_ADDRESS}
+      EPICS_PVA_ADDR_LIST: ${HOST_EXTERNAL_IP_ADDRESS}
+      EPICS_PVA_AUTO_ADDR_LIST: "NO"  
+      EPICS_PVA_ENABLE_IPV6: "false"  
+    command: >
+      /bin/bash -c "
+        java -DdefaultProtocol=pva -Dauthorization.permitall=false -Dauth.impl=demo -jar /saveandrestore/service-save-and-restore-*.jar"
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+volumes:
+  saveandrestore-es-data:
+    driver: local
+