Core udmi services integration stage two

bridgehead/README.md

@@ -0,0 +1,45 @@
+# Core UDMI Services Environment Set Up
+
+This guide provides instructions for deploying the core UDMI services bundle. This bundle includes:
+- Mosquitto broker
+- etcd server 
+- udmis service 
+- validator (registrar) tool 
+- pubber tool
+
+## Setting up the Docker environment
+
+1. **Install docker engine:** Ensure Docker is installed and running on your system: `https://docs.docker.com/engine/install/`
+
+2. **Navigate to project directory:** Open your terminal and change the directory to the project root containing the `docker-compose.yml` file (`bridgehead/`).
+
+3. **Get default site model:** In you terminal, run `git clone https://github.com/faucetsdn/udmi_site_model.git`.
+4. **Deploy the service:** Execute the following command to build the custom images (if needed) and start the containers in detached mode.
+    * **First time/after changes:** Run `sudo docker compose up -d --build`
+    * **Standard run:** Run `sudo docker compose up -d`
+5. **Confirm all containers are running:** Run `sudo docker ps` in the terminal, you should see the following containers in any order:
+   - pubber
+   - validator
+   - udmis
+   - mosquitto
+   - etcd
+
+## Tools
+
+The UDMI tools should only be run after the udmis service has completed setup. You can confirm this by comparing your udmis output to the [sample udmis output](sample_outputs/udmis_output.md). Tools should be run in the same directory as the Docker Compose (`bridgehead/`).
+
+### Registrar (validator)
+
+In your terminal, execute `sudo docker exec validator bin/registrar site_model/ //mqtt/mosquitto`. To confirm a successful execution, take a look at the [sample registrar output](sample_outputs/registrar_output.md)
+
+### Pubber
+
+The pubber tool will only have a successful output after the registrar tool has been executed. In your terminal, run `sudo docker exec pubber bin/pubber site_model/ //mqtt/mosquitto AHU-1 123456` (`123456` can be replaced with any serial number). 
+
+Pubber is running successfully if there are no obvious error messages or retries. An **unsuccessful** run will retry multiple times, will see messages like `Attempt #10 failed`. 
+
+A successful run will not end on its own, you can press `Ctrl` + `C` on your keyboard to exit. 
+
+## Shutting down the docker environment
+
+To gracefully stop and remove the container, run: `sudo docker compose down`

bridgehead/docker-compose.yml

@@ -0,0 +1,81 @@
+services:
+  mosquitto:
+    build:
+      context: ./mosquitto
+    container_name: mosquitto
+    volumes:
+      - ./udmi_site_model:/site_model
+      - ./var/mosquitto/certs:/etc/mosquitto/certs
+      - ./var/mosquitto/data:/etc/mosquitto/data
+      - ./var/mosquitto/log:/var/log/mosquitto
+    ports:
+      - "8883:8883"
+    networks:
+      - udminet
+
+  etcd:
+    image: quay.io/coreos/etcd:v3.5.13
+    container_name: etcd
+    volumes:
+      - ./var/etcd:/var/etcd
+    ports:
+      - "2379:2379"
+    networks:
+      - udminet
+    command: ["etcd", "-listen-client-urls=http://0.0.0.0:2379", "-advertise-client-urls=http://etcd:2379", "--data-dir", "/var/etcd"]
+
+  udmis:
+    build:
+      context: ./udmis
+    container_name: udmis
+    volumes:
+      - ./udmi_site_model:/root/site_model
+      - ./var/tmp:/tmp
+      - ./var/etcd:/root/udmi/var/etcd
+      - ./var/mosquitto/log:/var/log/mosquitto
+      - ./var/mosquitto/certs:/etc/mosquitto/certs
+    depends_on:
+      etcd:
+        condition: service_started
+      mosquitto:
+        condition: service_started
+    networks:
+      - udminet
+    environment:
+      ETCD_CLUSTER: etcd
+      MQTT_HOST: mosquitto
+      MQTT_PORT: 8883
+
+  validator:
+    image: ghcr.io/faucetsdn/udmi:validator-latest
+    container_name: validator
+    volumes:
+      - ./udmi_site_model:/root/site_model
+      - ./var/mosquitto/log:/var/log/mosquitto
+      - ./var/mosquitto/certs:/etc/mosquitto/certs
+    depends_on:
+      udmis:
+        condition: service_started
+    networks:
+      - udminet
+    command: ["tail", "-f", "/dev/null"]
+
+  pubber:
+    image: ghcr.io/faucetsdn/udmi:pubber-latest
+    container_name: pubber
+    volumes:
+      - ./udmi_site_model:/root/site_model
+    depends_on:
+      udmis:
+        condition: service_started
+    command: ["tail", "-f", "/dev/null"]
+    networks:
+      - udminet
+
+networks:
+  udminet:
+    name: udminet
+    driver: bridge
+    ipam:
+      config:
+        - subnet: 192.168.99.0/24

bridgehead/mosquitto/Dockerfile

@@ -0,0 +1,13 @@
+FROM ghcr.io/faucetsdn/udmi:udmis-latest AS udmi 
+FROM eclipse-mosquitto:latest
+
+RUN apk add --no-cache bash jq sudo git mosquitto mosquitto-clients openssl
+
+COPY --from=udmi /root/udmi/bin /root/udmi/bin  
+COPY --from=udmi /root/udmi/etc /root/udmi/etc  
+
+COPY mosquitto_startup.sh /usr/local/bin/
+RUN chmod +x /usr/local/bin/mosquitto_startup.sh
+
+CMD ["/usr/local/bin/mosquitto_startup.sh"]
+

bridgehead/mosquitto/mosquitto_startup.sh

@@ -0,0 +1,33 @@
+#!/bin/sh
+UDMI_ROOT=/root/udmi
+
+function fail {
+    echo error: $*
+    false
+}
+
+echo "persistence true
+persistence_location /var/lib/mosquitto/
+
+include_dir /etc/mosquitto/conf.d" > /etc/mosquitto/mosquitto.conf
+
+cd $UDMI_ROOT
+
+site_model=$(realpath /site_model)
+site_config=$site_model/cloud_iot_config.json
+registry_id=$(jq -r .registry_id $site_config)
+
+source $UDMI_ROOT/etc/mosquitto_ctrl.sh
+mkdir -p $CERT_DIR
+
+bin/setup_ca $site_model mosquitto  
+bin/start_mosquitto
+
+$MOSQUITTO_CTRL deleteClient $SERV_USER
+$MOSQUITTO_CTRL createClient $SERV_USER -p $SERV_PASS
+$MOSQUITTO_CTRL addClientRole $SERV_USER service
+
+echo Starting initializing site $site_model | tee -a $UDMIS_LOG
+bin/mosquctl_site $site_model
+
+sleep infinity

bridgehead/sample_outputs/registrar_output.md

@@ -0,0 +1,131 @@
+### Sample registrar service output for bridgehead docker setup 
+
+The registrar should only be run after the udmis service has been setup successfully (see the [sample udmis output](sample_outputs/udmis_output.md))
+
+The registrar has been run successfully if you see no errors and can spot the lines: 
+
+
+```
+Processed AHU-22 (3/4) in 0.029s (add)
+Processed SNS-4 (4/4) in 0.029s (add)
+Processed AHU-1 (2/4) in 0.102s (add)
+Processed GAT-123 (1/4) in 0.164s (add)
+```
+
+Sample output after running `sudo docker exec validator bin/registrar site_model/ //mqtt/mosquitto`:
+
+```
+starting run at 2025-11-05T10:12:02+00:00
+java -cp /root/validator/build/libs/validator-1.0-SNAPSHOT-all.jar com.google.daq.mqtt.util.Dispatcher registrar site_model/ //mqtt/mosquitto
+Writing reconciled configuration file to /root/out/registrar_conf.json
+Using reflector iot client
+Instantiating reflector client //mqtt/mosquitto/ ZZ-TRI-FECTA
+10:12:03.559 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - Loaded key /root/site_model/reflector/rsa_private.pkcs8 as sha256 42a8a8f1287aa775
+10:12:03.569 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - ZZ-TRI-FECTA token expiration sec 3600
+10:12:03.585 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - Using hash-key username/password /r/UDMI-REFLECT/d/ZZ-TRI-FECTA 42a8a8f1
+10:12:03.686 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - CA cert file: /root/site_model/reflector/ca.crt
+10:12:03.686 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - Device cert file: /root/site_model/reflector/rsa_private.crt
+10:12:03.686 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - Private key file: /root/site_model/reflector/rsa_private.pem
+10:12:03.686 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - Password sha256 0049165a
+10:12:03.703 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - ZZ-TRI-FECTA creating client /r/UDMI-REFLECT/d/ZZ-TRI-FECTA on ssl://mosquitto:8883
+10:12:03.917 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - ZZ-TRI-FECTA creating new auth token for audience mosquitto
+10:12:03.917 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - Using hash-key username/password /r/UDMI-REFLECT/d/ZZ-TRI-FECTA 42a8a8f1
+10:12:03.917 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - ZZ-TRI-FECTA connecting to mqtt server ssl://mosquitto:8883
+10:12:04.069 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - Subscribing with qos 1 to topic /r/UDMI-REFLECT/d/ZZ-TRI-FECTA/config
+10:12:04.071 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - Subscribing with qos 1 to topic /r/UDMI-REFLECT/d/ZZ-TRI-FECTA/errors
+10:12:04.072 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - Subscribing with qos 0 to topic /r/UDMI-REFLECT/d/ZZ-TRI-FECTA/commands/#
+10:12:04.073 [main] INFO  c.g.bos.iot.core.proxy.MqttPublisher - ZZ-TRI-FECTA done with setup connection
+Starting initial UDMI setup process
+Sending UDMI reflector state to ZZ-TRI-FECTA: {
+  "version" : "unknown",
+  "udmi" : {
+    "setup" : {
+      "udmi_version" : "unknown",
+      "functions_ver" : 18,
+      "udmi_commit" : "unknown",
+      "udmi_timever" : "unknown",
+      "msg_source" : "debug",
+      "tool_name" : "registrar",
+      "transaction_id" : "RC:92f139.00000001"
+    }
+  },
+  "timestamp" : "2025-11-05T10:12:03Z"
+}
+Received UDMI reflector initial config: {
+  "last_state" : "2025-11-05T10:12:03Z",
+  "reply" : {
+    "udmi_version" : "unknown",
+    "functions_ver" : 18,
+    "udmi_commit" : "unknown",
+    "udmi_timever" : "unknown",
+    "msg_source" : "debug",
+    "tool_name" : "registrar",
+    "transaction_id" : "RC:92f139.00000001"
+  },
+  "setup" : {
+    "hostname" : "5b24b7e56992",
+    "functions_min" : 18,
+    "functions_max" : 18,
+    "udmi_version" : "1.5.4-21-g61e7dba5",
+    "udmi_commit" : "61e7dba58",
+    "udmi_ref" : "ghcr.io/faucetsdn/udmi:udmis-g61e7dba5",
+    "udmi_timever" : "2025-11-04T13:20:19Z",
+    "built_at" : "2025-11-04T13:30:00Z",
+    "built_by" : "runner@runnervmf2e7y"
+  }
+}
+Subscribed to /r/UDMI-REFLECT/d/ZZ-TRI-FECTA
+Instantiated iot provider mqtt as IotReflectorClient
+Working with project mosquitto registry us-central1/ZZ-TRI-FECTA
+Loading site_defaults.json
+Initializing 4 local devices...
+Starting initialize settings for 4 devices...
+Waiting 61s for 4 tasks to complete...
+Fetching devices from registry ZZ-TRI-FECTA...
+2025-11-05T10:12:04Z Fetched 0 devices.
+Fetched 0 device models from cloud registry
+Processing 4 new devices...
+Waiting for device processing...
+Waiting 61s for 4 tasks to complete...
+Processed AHU-22 (3/4) in 0.029s (add)
+Processed SNS-4 (4/4) in 0.029s (add)
+Processed AHU-1 (2/4) in 0.102s (add)
+Processed GAT-123 (1/4) in 0.164s (add)
+Processed 4 (skipped 0) devices in 0.823s, 0.205s/d
+Updating 0 existing devices...
+Waiting for device processing...
+Processed 0 (skipped 0) devices in 0.000s, 0.000s/d
+Updated 0 device metadata files.
+Finished processing 4/4 devices.
+Binding devices to gateways: [GAT-123]
+Waiting for device binding...
+Waiting for tasks to complete...
+Already bound to GAT-123: []
+Binding [AHU-22, SNS-4] to GAT-123 (1/1)
+Finished binding gateways in 0.151
+Starting writing normalized for 4 devices...
+Waiting 61s for 4 tasks to complete...
+Writing normalized /root/site_model/devices/AHU-22/out/metadata_norm.json
+Writing normalized /root/site_model/devices/GAT-123/out/metadata_norm.json
+Writing normalized /root/site_model/devices/AHU-1/out/metadata_norm.json
+Writing normalized /root/site_model/devices/SNS-4/out/metadata_norm.json
+Starting previewing model for 4 devices...
+Waiting 61s for 4 tasks to complete...
+Starting validating expected for 4 devices...
+Waiting 61s for 4 tasks to complete...
+Starting validate samples for 4 devices...
+Waiting 61s for 4 tasks to complete...
+Starting validating keys for 4 devices...
+Waiting 61s for 4 tasks to complete...
+Starting process externals for 4 devices...
+Waiting 61s for 4 tasks to complete...
+Updating errors site_model/devices/AHU-1/out/errors.map
+
+Summary:
+  Device envelope: 1
+  Device status: 4
+  Device validation: 1
+Out of 4 total.
+Registration summary available in /root/site_model/out/registration_summary.json
+Registration summary available in /root/site_model/out/registration_summary.csv
+```

bridgehead/sample_outputs/udmis_output.md

@@ -0,0 +1,58 @@
+### Sample udmis service output for bridgehead docker setup 
+
+The udmis service first waits 15 seconds for the mosquitto container to setup, and then takes under 60 seconds to set itself up. You'll know the service is up and running when you see the line: `udmis running in the background, pid XX log in /tmp/udmis.log`. Look for that exact line and no big errors above it (The process ID, or PID, shown as XX, will be a different number every time you run the service). Below shows the command you should run to see your log, as well as a sample successful udmis log output.
+
+You should run the following command in the same directory (`/bridgehead`) where you ran the initial docker compose command: `sudo docker logs udmis -f`
+
+Once you have confirmed the udmis service is running, press `Ctrl` + `C` on your keyboard to exit the log view and return to your terminal prompt.
+
+Sample output:
+```
+~ # cat /usr/local/bin/startup_output.txt 
+Waiting 15s for mqtt to setup
+fatal: not a git repository (or any of the parent directories): .git
+fatal: not a git repository (or any of the parent directories): .git
+fatal: not a git repository (or any of the parent directories): .git
+fatal: not a git repository (or any of the parent directories): .git
+fatal: not a git repository (or any of the parent directories): .git
+Starting local services at 2025-11-05T09:24:26+00:00
+Starting udmis proper...
+Waiting for udmis startup 29...
+Waiting for udmis startup 28...
+Waiting for udmis startup 27...
+Waiting for udmis startup 26...
+Waiting for udmis startup 25...
+::::::::: tail /tmp/udmis.log
+2025-11-05T09:24:31.461Z xxxxxxxx N: MessageDispatcherImpl Scheduling provision execution, after 60s every 60s
+2025-11-05T09:24:31.462Z xxxxxxxx I: MessageDispatcherImpl dispatcher/provision activating provision with 3e7dfd63
+2025-11-05T09:24:31.462Z xxxxxxxx D: SimpleMqttPipe Activating message pipe provision as 0000001f => 31
+2025-11-05T09:24:31.462Z xxxxxxxx N: SimpleMqttPipe Creating new source queue provision with capacity 1000
+2025-11-05T09:24:31.462Z xxxxxxxx D: SimpleMqttPipe Handling provision
+2025-11-05T09:24:31.462Z xxxxxxxx I: SimpleMqttPipe Starting message loop 723ed5a0:00
+2025-11-05T09:24:31.462Z xxxxxxxx I: SimpleMqttPipe Starting message loop 723ed5a0:02
+2025-11-05T09:24:31.462Z xxxxxxxx I: SimpleMqttPipe Starting message loop 723ed5a0:01
+2025-11-05T09:24:31.463Z xxxxxxxx I: SimpleMqttPipe Starting message loop 723ed5a0:03
+2025-11-05T09:24:31.463Z xxxxxxxx I: SimpleMqttPipe Subscribed mqtt-295ac31c to topic /r/+/d/+/events/discovery
+2025-11-05T09:24:31.463Z xxxxxxxx I: StateProcessor Activating
+2025-11-05T09:24:31.463Z xxxxxxxx N: StateProcessor Scheduling state execution, after 60s every 60s
+2025-11-05T09:24:31.463Z xxxxxxxx D: MessageDispatcherImpl Registering handler for Object in dispatcher/state
+2025-11-05T09:24:31.463Z xxxxxxxx D: MessageDispatcherImpl Registering handler for Exception in dispatcher/state
+2025-11-05T09:24:31.464Z xxxxxxxx D: MessageDispatcherImpl Registering handler for StateUpdate in dispatcher/state
+2025-11-05T09:24:31.464Z xxxxxxxx I: MessageDispatcherImpl Activating
+2025-11-05T09:24:31.464Z xxxxxxxx N: MessageDispatcherImpl Scheduling state execution, after 60s every 60s
+2025-11-05T09:24:31.464Z xxxxxxxx I: MessageDispatcherImpl dispatcher/state activating state with 3d36e013
+2025-11-05T09:24:31.464Z xxxxxxxx D: SimpleMqttPipe Activating message pipe state as 0000001f => 31
+2025-11-05T09:24:31.464Z xxxxxxxx N: SimpleMqttPipe Creating new source queue state with capacity 1000
+2025-11-05T09:24:31.464Z xxxxxxxx D: SimpleMqttPipe Handling state
+2025-11-05T09:24:31.464Z xxxxxxxx I: SimpleMqttPipe Starting message loop 7abe27de:00
+2025-11-05T09:24:31.465Z xxxxxxxx I: SimpleMqttPipe Starting message loop 7abe27de:01
+2025-11-05T09:24:31.465Z xxxxxxxx I: SimpleMqttPipe Starting message loop 7abe27de:02
+2025-11-05T09:24:31.465Z xxxxxxxx I: SimpleMqttPipe Starting message loop 7abe27de:03
+2025-11-05T09:24:31.465Z xxxxxxxx I: SimpleMqttPipe Subscribed mqtt-89bd90c3 to topic /r/+/d/+/state
+2025-11-05T09:24:31.465Z xxxxxxxx I: PubSubIotAccessProvider Activating
+2025-11-05T09:24:31.466Z xxxxxxxx I: PubSubIotAccessProvider Fetching registries for: 
+2025-11-05T09:24:31.466Z xxxxxxxx D: PubSubIotAccessProvider Fetched 0 registry regions
+2025-11-05T09:24:31.466Z xxxxxxxx N: UdmiServicePod Finished activation of container components, created /tmp/pod_ready.txt
+udmis running in the background, pid 39 log in /tmp/udmis.log
+Blocking until termination.
+```

bridgehead/udmis/Dockerfile

@@ -0,0 +1,13 @@
+FROM quay.io/coreos/etcd:v3.5.13 AS etcd-client-builder
+FROM ghcr.io/faucetsdn/udmi:udmis-latest
+
+RUN apk add --no-cache bash openjdk21 gcompat curl jq sudo git python3 moreutils mosquitto mosquitto-clients openssl coreutils nano
+
+COPY --from=etcd-client-builder /usr/local/bin/etcdctl /usr/local/bin/etcdctl
+COPY startup/mosquitto_ctrl.sh /root/udmi/etc/
+COPY startup/udmis_startup.sh /usr/local/bin/startup/
+
+RUN chmod +x /usr/local/bin/startup/udmis_startup.sh
+RUN chmod +x /root/udmi/etc/mosquitto_ctrl.sh
+
+CMD ["/usr/local/bin/startup/udmis_startup.sh"]