Alarm server

[Lerb123]

Good morning everyone, I am new to Phoebus, could someone help me with the following:

I am trying to get the Phoebus alarm server running. From the terminal everything works fine for me, no problem. The thing is when I want to start from the “alarm_server.service” (this file is in app/alarm/examples). I want that every time the service starts when starting the machine, it detects a specific alarm configuration with its components and associated pvs, but when I do it, it only detects the configuration but the server runs for a few seconds and then crashes.

What can I be doing wrong, or if anyone has any example of how I should configure it to start well with its configuration from an xml file and stay active?

[georgweiss]

Do you have a stack trace from the crash?

[Lerb123]

I use Ubuntu desktop xD

[Lerb123]

And I'm using the alarm_server.service from Phoebus/app/alarm/examples

[georgweiss]

But you're saying the service crashes after a while. To figure out why a log of some kind is essential.

[Lerb123]

I am already at work. I attach how I have the service configured:

[Unit]
Description=Phoebus Alarm Server
After=kafka.service

[Service]
User=administrador
Group=administrador
Environment=JAVA_HOME=/usr/lib/jvm/bellsoft-java21-amd64
Environment=SERVER=/home/administrador/git/forks/phoebus/services/alarm-server
Environment=PORT=4609
Environment=CONFIG=CCBna_01
ExecStart=/usr/bin/procServ --foreground --noautorestart --name alarm-server --chdir ${SERVER} --logfile ${SERVER}/console.log ${PORT} ./alarm-server.sh -import /home/administrador/git/forks/phoebus/services/alarm-server/config.xml -config ${CONFIG}

[Install]
WantedBy=multi-user.target

And this is what it shows me in the console.log:

@@@ Restarting child "alarm-server"
@@@ (as ./alarm-server.sh)
@@@ The PID of new child "alarm-server" is: 3636
@@@ @@@ @@@ @@@ @@@
2025-02-05 05:23:20 INFORMACIÓN [org.phoebus.applications.alarm] Import model from /home/administrador/git/forks/phoebus/services/alarm-server/config.xml
Reading new configuration from /home/administrador/git/forks/phoebus/services/alarm-server/config.xml
Fetching existing alarm configuration for "CCBna_01" with 10 connection timeout, then waiting for it to remain stable for 4 seconds...
2025-02-05 05:23:20 INFORMACIÓN [org.phoebus.applications.alarm] Reading from start of CCBna_01
Deleting existing CCBna_01 ...
Loading new CCBna_01 ...
Time to read XML : 00:00:00.068
Time to fetch existing configuration : 00:00:10
Time to delete existing configuration: 00:00:00.001
Time to load new configuration : 00:00:00.205
Total : 00:00:10
Done.
2025-02-05 05:23:30 INFORMACIÓN [org.phoebus.applications.alarm] AlarmClientModel CCBna_01 shut down

@@@ @@@ @@@ @@@ @@@
@@@ Received a sigChild for process 3636. Normal exit status = 0
@@@ Current time: Wed Feb 5 05:23:30 2025
@@@ Child process is shutting down, auto restart is disabled
@@@ ^R or ^X restarts the child, ^Q quits the server

[Lerb123]

Topic: CCBna_01 (example)

config:/CCBna_01/Amoniaco {"user":"administrador","host":"optiplex7040","commands":[{"title":"Area1 Command Title 1","details":"Area1 Command Detail 1"},{"title":"Area1 Command Title 2","details":"Area1 Command Detail 2"}],"actions":[{"title":"Area3: Amoniaco","details":"Area1 Action Detail 1","delay":5},{"title":"Area1 Action Title 2","details":"Area1 Action Detail 2","delay":6}]}
config:/CCBna_01/Amoniaco/presion {"user":"administrador","host":"optiplex7040","description":"a1pv1 description","delay":4,"count":5,"filter":"a1pv1 filter","guidance":[{"title":"a1pv1 Guidance Title 1","details":"a1pv1 Guidance Detail 1"}],"displays":[{"title":"a1pv1 Display Title 1","details":"a1pv1 Display Detail 1"}],"commands":[{"title":"a1pv1 Command Title 1","details":"a1pv1 Command Detail 1"}]}
config:/CCBna_01/Carbonatador {"user":"administrador","host":"optiplex7040","commands":[{"title":"Area1 Command Title 1","details":"Area1 Command Detail 1"},{"title":"Area1 Command Title 2","details":"Area1 Command Detail 2"}],"actions":[{"title":"Area 4: Linea 2 Carbonatador","details":"Area1 Action Detail 1","delay":5},{"title":"Area1 Action Title 2","details":"Area1 Action Detail 2","delay":6}]}
config:/CCBna_01/Carbonatador/carbonatador {"user":"administrador","host":"optiplex7040","description":"a1pv1 description","delay":4,"count":5,"filter":"a1pv1 filter","guidance":[{"title":"a1pv1 Guidance Title 1","details":"a1pv1 Guidance Detail 1"}],"displays":[{"title":"a1pv1 Display Title 1","details":"a1pv1 Display Detail 1"}],"commands":[{"title":"a1pv1 Command Title 1","details":"a1pv1 Command Detail 1"}]}
config:/CCBna_01/Linea 1 {"user":"administrador","host":"optiplex7040","commands":[{"title":"Area1 Command Title 1","details":"Area1 Command Detail 1"},{"title":"Area1 Command Title 2","details":"Area1 Command Detail 2"}],"actions":[{"title":"Area: Linea 1","details":"Area perteneciente a la Linea 1","delay":5},{"title":"Area1 Action Title 2","details":"Area1 Action Detail 2","delay":6}]}
config:/CCBna_01/Linea 1/speed {"user":"administrador","host":"optiplex7040","description":"velocidad de las cintas transportadoras","latching":false,"annunciating":false,"delay":4,"count":5,"filter":"a1pv1 filter","guidance":[{"title":"a1pv1 Guidance Title 1","details":"a1pv1 Guidance Detail 1"}],"displays":[{"title":"a1pv1 Display Title 1","details":"a1pv1 Display Detail 1"}],"commands":[{"title":"a1pv1 Command Title 1","details":"a1pv1 Command Detail 1"}]}
config:/CCBna_01/Linea 2 {"user":"administrador","host":"optiplex7040","commands":[{"title":"Area1 Command Title 1","details":"Area1 Command Detail 1"},{"title":"Area1 Command Title 2","details":"Area1 Command Detail 2"}],"actions":[{"title":"Area2: Linea 2","details":"Area pertenece a Linea 2","delay":5},{"title":"Area1 Action Title 2","details":"Area1 Action Detail 2","delay":6}]}
config:/CCBna_01/Linea 2/sim://sine {"user":"administrador","host":"optiplex7040","description":"a1pv1 description","delay":4,"count":5,"filter":"a1pv1 filter","guidance":[{"title":"a1pv1 Guidance Title 1","details":"a1pv1 Guidance Detail 1"}],"displays":[{"title":"a1pv1 Display Title 1","details":"a1pv1 Display Detail 1"}],"commands":[{"title":"a1pv1 Command Title 1","details":"a1pv1 Command Detail 1"}]}
config:/CCBna_01/Linea 2/temperature {"user":"administrador","host":"optiplex7040","description":"a1pv1 description","delay":4,"count":5,"filter":"a1pv1 filter","guidance":[{"title":"a1pv1 Guidance Title 1","details":"a1pv1 Guidance Detail 1"}],"displays":[{"title":"a1pv1 Display Title 1","details":"a1pv1 Display Detail 1"}],"commands":[{"title":"a1pv1 Command Title 1","details":"a1pv1 Command Detail 1"}]}

[Lerb123]

I noticed that when the “-import” parameter is set, the “AlarmClientMode” closes, and the server is not active, which is strange. Shouldn't it stay active if I configure it to see a configuration and open it?

I thought about just setting the "-import", but if I do it shows this:

2025-02-05 05:52:23 INFO [org.phoebus.applications.alarm] Import model from config.xml
Reading new configuration from config.xml
Expecting configuration for "Accelerator" but file config.xml contains settings for "CCBna_01".

If I set both parameters “import ” “ config” , the alarm client dies and does not remain active.

If I set only the “-config 'topicName' ” I open the topic in phoebus but empty (no components or pvs).

[Lerb123]

Yes, I built my topic as follows

[kasemir]

You should then be able to stop kafka & zookeeper, restart them, and monitor_topic.sh again shows the config.
When you now start the alarm GUI, it will show the config.
Finally, you start the alarm server, and now the state of the config will update as PVs listed in the config go in and out of alarm.

[Lerb123]

Thanks kasemir and George, everything is working fine 😊.

[kasemir]

What was the main problem? Why did your configuration appear empty?

[Lerb123]

I had an error in one of the environment variables of the alarm server, and I was doing one of the steps in the tutorial wrong

[kasemir]

-import and -export do just that and then exit.
To run, you start it with just the -config to run.

[kasemir]

I am trying to get the Phoebus alarm server running.

Have you tried following the steps in https://github.com/ControlSystemStudio/phoebus/blob/master/app/alarm/Readme.md ?

[limesTech]

Hi,
similar to Lerb123 I'm very new to this community and trying to set up an alarm server for a different site.
Can anybody hint me to a documentation on the 'ALARM_SERVICE_SETTINGS_FILE' and the 'SETTINGS_FILE' mentioned in the docker compose file. The current pages alarm server doc is very minimal and outdated.

[shroffk]

The alarm server doc you linked are pointing to high level explanination of the service.

https://github.com/ControlSystemStudio/phoebus/blob/master/app/alarm/Readme.md
Is closer to the installation instructions... we will update these too ( along with the docker images ) I hope you can help us test the new documentations

[georgweiss]

I realize the docker compose files are not up to standards. Will update these and add suitable information.

[shroffk]

Thanks @georgweiss

[georgweiss]

@limesTech, README and compose files updated. Please let me know if you have any questions.

[limesTech]

@georgweiss thanks for the additional link in the README.

In future, a hint on the settings.ini would be nice as well. The Problem org.phoebus.applications.alarm/config_names= needs to be a single element could cause confusion. I tried reusing the settings.ini from our Phoebus fontend, but sadely this is not possible with this list loading issue.

Does anyone still maintain: control-system-studio.readthedocs.io other than adding the GitHub README Links? If not I would update it, after I finished our Alarm system setup at our facility.

[limesTech]

@georgweiss is there any reason why services/docker-compose-kafka-elastic-mongodb.yml still uses Zookeeper instead of KRaft?

Performance, lack of testing, or does Elasticsearch/Alarmlogger ... depend on Zookeeper specific functions?

[georgweiss]

No other reason than "legacy". Have not yet had the time to explore a non-Zookeper scenario.

[limesTech]

Ok thx

[kasemir]

Lack of testing, see #3441 for "basically, we think it's fine"

[limesTech]

In my setup, the Alarmserver and the Kafka(+ KRaft) docker container are running on different hosts. I'm currently wondering why the Alarmserver Application does not connect to Kafka properly. It does not create (new) Topics in Kafka and if created manually it still shuts down with the following Error messages:

/VacuumAlarm > 
VacuumAlarmServ  | End of input, exiting command shell.
VacuumAlarmServ  | 2025-07-14 17:57:51 INFO [org.phoebus.applications.alarm] Shutting down
VacuumAlarmServ  | 2025-07-14 17:57:51 WARNING [org.phoebus.applications.alarm] Error reading updates from Kafka
VacuumAlarmServ  | org.apache.kafka.common.errors.WakeupException
VacuumAlarmServ  |      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeTriggerWakeup(ConsumerNetworkClient.java:529)
VacuumAlarmServ  |      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:293)
VacuumAlarmServ  |      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:251)
VacuumAlarmServ  |      at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1255)
VacuumAlarmServ  |      at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1186)
VacuumAlarmServ  |      at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159)
VacuumAlarmServ  |      at org.phoebus.applications.alarm.server.ServerModel.checkUpdates(ServerModel.java:214)
VacuumAlarmServ  |      at org.phoebus.applications.alarm.server.ServerModel.run(ServerModel.java:146)
VacuumAlarmServ  |      at java.base/java.lang.Thread.run(Unknown Source)
VacuumAlarmServ  | 
VacuumAlarmServ  | 2025-07-14 17:57:51 INFO [org.phoebus.applications.alarm] ServerModel shut down
VacuumAlarmServ  | 2025-07-14 17:57:51 INFO [org.phoebus.applications.alarm] Stopped all PVs
VacuumAlarmServ  | 2025-07-14 17:57:51 WARNING [org.apache.kafka.clients.NetworkClient] [Producer clientId=producer-1] Error while fetching metadata with correlation id 103 : {VacuumAlarm=UNKNOWN_TOPIC_OR_PARTITION, VacuumAlarmTalk=UNKNOWN_TOPIC_OR_PARTITION}

If there is no correctly named Topic, the producer creates a lot of errors until it times out like below

| 2025-07-14 18:55:00 WARNING [org.apache.kafka.clients.NetworkClient] [Producer clientId=producer-1] Error while fetching metadata with correlation id 201 : {VacuumAlarm=UNKNOWN_TOPIC_OR_PARTITION, VacuumAlarmTalk=UNKNOWN_TOPIC_OR_PARTITION}
VacuumAlarmServ  | 2025-07-14 18:55:00 INFO [org.phoebus.applications.alarm] Cleared configuration for VacuumAlarm

I did test to create topics from outside the network via the Kafka .sh scripts and it works perfectly.
(If I run the Alarmserver alongside Kafka on the same machine, everything is working.)
Firewalls were ruled out. It looks like Kafka does recognize the Alarmserver and assigns him a group ID and after ~10s Kafka receives an explicit LeaveGroup request and the consumer is being closed. And in the Kafka-UI I found some {SEVERITY: OK} messages, so there was a message.

Any guesses on what I'm missing ?

Is this maybe an intended behavior, if no PVs are available in the network?

[kasemir]

It does not create (new) Topics in Kafka

It shouldn't. See https://github.com/ControlSystemStudio/phoebus/blob/master/app/alarm/Readme.md comments about auto.create.topics.enable=false and create_alarm_topics.sh.

If there is no correctly named Topic ... UNKNOWN_TOPIC_OR_PARTITION

That at least means it is connecting alright. Once you create the topics, that error should go away.

shuts down with the following Error messages:

/VacuumAlarm > 
VacuumAlarmServ  | End of input, exiting command shell.
VacuumAlarmServ  | 2025-07-14 17:57:51 INFO [org.phoebus.applications.alarm] Shutting down

"End of input" looks like you're typing "exit" or "Control-D" in the alarm server terminal to stop it on purpose.
If you're not running in a terminal or under procServ, you need to disable the terminal via -noshell. But then you won't have a terminal to check things out on the server side.

[limesTech]

@kasemir Thanks, I was obviously too tiered, to see the obvious. There was an auto.create config change, and the end of a docker cmd command is equal to an exit input.

I will keep the auto.create enabled, until I find a feasible solution for creating Kafka topic outside the Kafka host.

Or do other facilities have a different approach on how to deploy different alarmservers with arbitrary Topics after deploying Kafka?

Dislike the idea of keeping a copy of Kafka at every alarmserver container, only to run the create_alarm_topics.sh as pre-up script.

[shroffk]

Our alarm services and kafka run as separate services (sometimes on different VMs or containers).
When deploying an alarm server we use ansible to run the create topics script mentioned above to set up the appropriate topics for each instance of the alarm server we are running.

Dislike the idea of keeping a copy of Kafka at every alarmserver container, only to run the create_alarm_topics.sh as pre-up script.

I am not sure I understand the limitations which would require an instance of kafka for each alarm server in the same container.

[kasemir]

will keep the auto.create enabled

You can't run like that!
We rely on the alarm status topic to be 'compacted'. We don't want kafka to keep the complete history of all alarm states, because then a newly started GUI would replay to you what happened in the last couple years.
We need kafka to 'compact' the topics to just show the current state, maybe the last few updates to get to the current state.
See basic alarm system design, https://accelconf.web.cern.ch/icalepcs2019/papers/wesh2001.pdf

If you auto-create the topics, they have some default configuration that'll be wrong.

[limesTech]

@kasemir Uff, I see, so just enforcing cleanup.policy=compact on auto.create will not solve the problem because you depend on the creation of these different types of topics.
@shroffk I guess with shared pools or Ansible is the only way

[limesTech]

Is there an Alarmserver endpoint to check whether the server/container is healthy ?

[shroffk]

I think this would be nice addition
We do have the heartbeat which is used by the Phoebus clients but a service endpoint would be very useful.

[jacomago]

Maybe we should add actuator to all the spring boot applications (its in channelfinder already) it includes a health end point amongst other metrics to check on.