YAML File Structures

YAML File Structures

This is an example of a minimal Hyperion system configuration:

name: Example-Config
env: env/example_env.sh

groups:
- name: Example Group 1
  components:
  - name: xclock
    cmd:
    - start: xclock
    - check: pgrep xclock > /dev/null
    host: example-host-1

The above example describes a system named Example-config with a custom environment file example_env.sh at a relative path. It contains only one component group named Example Group 1 which holds a single component named xclock to be run on a host referenced by hostname example-host-1. On start the component will run the shell command xclock and the check command of the component searches for a xclock process piping the output to /dev/null.

The mandatory fields of a system configuration are the name and a list of groups holding at least one group that contains a single component. The only optional field is the env entry, that holds the path to a custom environment file (both relative to the system file or absolute path will work) which is sourced before component commands and for parsing environment variables in the configuration.

A component group consists of a (preferably short) name and a list of components. It has no optional fields yet.

The component configurations are the files that hold the most information about system configurations: It requires a name, a command list (cmd) and a host to be valid. The only mandatory entry in thecmd list is start, which defines the command to be run on component startup. A check command can be provided that will be run as second stage of a component check (first stage is checking if the child process of the tmux window (the invoked command) is still running). Note that currently this is the only part of the configuration where the order matters: start needs to be the first and check needs to be the second element of the list! Further, as described in #34, an optional stop command to be run on component stop will come soon.

Multiline commands are also supported by the yaml format, you only have to use folding style or literal style.

An optional top-level field of a component is wait. It defines the time in seconds to wait after a component start for a successful component check until it is considered to have failed (note that a component start will return as soon as it receives a successful check status or the component wait time is reached). The default value used for wait is defined in config.py (currently 3 seconds).

The next optional field is the depends entry: it is a list of components that need to be running in order for this component to start. Note: Here you have to provide component ids (COMPONENT_NAME@COMPONENT_HOST). Component ids, like the name suggests, are used to identify components. If you are wondering now, why you don't have to specify a component id, you already gave all the required information and the parser will add this field automatically at runtime.

Since 13f0cb5 it is possible to use environment variables as hostnames to make the configuration more dynamic. You only need to use ${your_env_var} and the parser will replace it with the value in your environment (both the host and the depends entries support this).

The last optional field noauto was added in 5712297. It marks the component to never start unless it specifically is instructed by the user to do so. This can be used for tool components that are not necessary for a system to run, but still are handy to have in the user interface for occasional quick access (teleoperation, sim control, ...). To mark a component with noauto you simply have to add the field to the file. A value is not required but adding some symbolic value (like True) could enhance clarity of what is happening there.
Note: It is highly discouraged to make any component dependent on a component marked with noauto, because it will mess up the startup process (especially if the dependent component is not marked with noauto: the start_all procedure cannot be successful in that case!)

Component Example

name: Component name
cmd:
  - start: start shell command
  - check: |
      This is a literal style multiline
      check shell command
      If this does not work, check the indentation levels
depends:
  - dependency1 (component id) 
  - ...
wait: N seconds 
noauto: This value is not necessary
host: hostname or ${env_var}

ATTENTION

Tee is used to redirect the stdout and stderr of a component process to a log file, which causes to change the buffering mode of the filedescriptors, thus if buffering is not specified by the executed command, the output will not be buffered by line. This also changes the buffering on the tty (output is only shown after the process finished running) which is bad if you want to keep track of a components state by taking a look at the stdout at runtime. This can be circumvented by adding stdbuf -oL -eL in front of your (last) start command (this will enforce linebuffering for stdout and stderr). This command is included in GNU coreutils so it is preinstalled on almost any linux machine.
(Issue #38 tackles this problem)

ATTENTION

The parser allows splitting configurations in multiple files, which not only enhances clarity but also allows reusing components for different system configurations.

To include a yaml file inside another yaml file use the !include relative/path/to/file.yaml markup. Sadly pyyaml needs the include to be inside a field to allow appending fields in the including file. So something like this won't work:

!include nested.yaml
name: ¯\_(ツ)_/¯

I came up with a structure splitted to up to four different file types: systems, groups, components and cores. The idea is to keep those organized with the following directory structure:

.
├── components
│   ├── cores
│   │   ├── Core_0.yaml
│   │   ├── Core_1.yaml
│   │   ├── ...
│   │   └── Core_N.yaml
│   ├── groups
│   │   ├── Group_0.yaml
│   │   ├── Group_1.yaml
│   │   ├── ...
│   │   └── Group_N.yaml
│   ├── Component_0.yaml
│   ├── Component_1.yaml
│   ├── ...
│   └── Component_N.yaml
└── systems
    ├── system_0.yaml
    ├── system_1.yaml
    ├── ...
    └── system_N.yaml 

The insides of system and group files should be clear, but I'll go more into detail with component and core files:

Imagine that you have a scenario involving a robot and you want to have a configuration for the real world system on the robot, but also one for to run a simulation of the robot on an arbitrary PC and you wan't to reuse as much configuration as you can.
Many configuration aspects like filepaths can be handled by using conditional exports in a custom environment file, but when it comes to component dependency structures you need to have different files. This is where split definitions in core files come in handy. Inside those you just define the contents of the cmd field.

example_core.yaml

- start: |
    if [ $basepc == "robot" ] ; then
        robot_envros
        stdbuf -oL -eL roslaunch robot_bringup robot_default.launch ${BRINGUP_ARGS}
    else
        envros
        stdbuf -oL -eL roslaunch robot_bringup robot_default.launch ${BRINGUP_ARGS} robot_sim_on:=true
    fi
- check: |
    source "${prefix}/setup.bash"
    rostopic info /robot/odom | grep "Publishers" | grep -q -v "None"

This core is able to run on the robot or the simulation by adapting its start command to the current environment. But while on the real robot this core may only needs a roscore to run, on a PC running a simulation additionally it needs the simulation to be running and the robot to be spawned. So we create two component files including the core, but defining different dependencies and along with the component name and the host to run on.

Example_split_robot.yaml

name: bringup
cmd: !include cores/example_core.yaml
depends:
  - roscore@${robot_pc}
host: ${robot_pc}

Example_split_sim.yaml

name: bringup
cmd: !include cores/example_core.yaml
depends:
  - sim-ros-bridge@${robot_pc}
host: ${robot_pc}

Note: The host they run on is defined by the same environment variable, but depending on the custom environment this could evaluate to different values on different hosts.

Now that we have those two component files, the file including this component needs to be split too, so depending on your structure you need to make two group files and two system files with different includes.
Now you have a shared component cmd, so you won't have to apply the same changes in two files.

Sidenote

Because of the 'no top-level include with appending fields' issue we have to declare some top-level fields twice that I would have liked to include in the core. If you find a way of doing this or you happen to have developed a better structure don't hesitate to let me know :)